A workflow utilizing general-purpose large language models for efficient structuring and data mining of bone scintigraphy narratives Authors: WenXin Tang, Jun Zhou, Wenxin Chen, Yunze Xie, Lichao Zhang, Shuting Chen, Xiaoguang Hou, Shuo Wang, Hongcheng Shi Published: 2026 ABSTRACT AbstractWhole-body bone scintigraphy is pivotal for skeletal evaluation in oncological monitoring, yet the unstructured nature of clinical reports impedes efficient data management and multicenter integration. This study aims to validate whether a clinical-logic-guided prompting framework can effectively constrain general-purpose large language models (LLMs) to achieve reliable structured information extraction from bone scintigraphy narratives without domain-specific fine-tuning, and to evaluate the performance of this workflow in real-world clinical scenarios. We first established a multicenter ground-truth dataset to benchmark four LLMs (DeepSeek-R1, DeepSeek-V3, GPT-o3, and Gemini 2.5 Pro), and DeepSeek-R1 demonstrated the highest accuracy and stability in structured extraction. Subsequently, using DeepSeek-R1, we executed two validation tasks. In a human-in-the-loop workflow, LLM assistance reduced manual processing time by 74.5%-82.6% while significantly enhancing accuracy. Furthermore, we constructed a bone metastasis atlas for eight common malignancies through the automated processing of data from initial 18,331 patients. Our study demonstrates that prompt engineering designed by clinical experts, integrating clinical logic and controlled vocabularies, can effectively guide general-purpose LLMs for bone scintigraphy narrative information extraction. This approach provides a validated low-code paradigm for physicians to transform medical narratives into analyzable structured data, thereby empowering large-scale clinical research.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-026-50745-2.Keywords: Bone metastasis, Artificial intelligence, LLMs, RadiologySubject terms: Cancer, Computational biology and bioinformatics, Health care, Mathematics and computing, Medical research, Oncology Introduction Bone metastasis is a common complication in many cancers, with distinct malignancies showing a marked preference for specific skeletal sites. Understanding these metastatic patterns is crucial for prognostic stratification, treatment planning, and follow-up monitoring1,2. Whole-body bone scintigraphy remains a cornerstone imaging technique for systemic skeletal evaluation due to its high sensitivity, availability, and cost-effectiveness3–5. Therefore, bone scintigraphy reports serve as essential documents for guiding clinical decisions. Furthermore, large-scale retrospective analysis of these reports allows physicians to better understand the metastatic patterns of bone involvement across different cancers. However, a significant gap exists between clinical documentation and data utilization. These imaging reports are predominantly recorded as narrative text documents, characterized by variations in terminology and phrasing across different time periods, medical institutions, and physicians6. This heterogeneity creates a double burden for physicians. Firstly, in daily practice, manual structuring of reports for patient tracking is labor-intensive and inefficient. Secondly, for research, extracting structured data from massive amounts of narrative text is prohibitively costly. Conventional automated extraction methods often struggle with accuracy, failing to meet the rigorous demands of both clinical practice and large-scale research. Large language models (LLMs) have demonstrated exceptional performance in tasks such as medical diagnosis and case classification, underscoring their potential for processing specialized medical knowledge7–11. Early studies have assessed the capacity of LLMs to extract information from medical narratives and have validated their utility in select clinical settings12–15. Despite this promise, widespread adoption remains difficult. Research addressing domain-specific tasks frequently relies on resource-intensive methods such as model fine-tuning, which creates a high technical barrier for physicians and researchers without computational expertise16. Reasoning LLMs, such as DeepSeek-R1, offer a promising alternative, potentially using chain-of-thought capabilities to adhere to complex clinical rules and constraints without the need for extensive training. By leveraging LLMs’ chain-of-thought capabilities and incorporating specific rules and requirements for clinical structured reporting into prompts, clinicians can utilize LLMs for structured information extraction from narratives in a low-code, low-threshold manner. We hypothesized that integrating clinical logic and controlled vocabularies into prompts would constrain general-purpose LLMs to achieve reliable information extraction from narrative bone scintigraphy reports. Beyond extraction accuracy, we also anticipated that this framework would improve clinician efficiency in human-in-the-loop workflows and support the construction of a bone metastasis atlas through automated extraction. To test these hypotheses, we developed a clinical-logic-guided framework, evaluated its performance across multiple LLMs, and validated its utility through clinical efficiency analysis and large-scale data processing. Materials and methods Study design and data source This retrospective, multicenter study analyzed clinical narratives (bone scintigraphy reports and relevant medical history) collected between January 1, 2016, and December 31, 2023. Data were sourced from three institutions: Zhongshan Hospital, Fudan University (ZS); Shanghai Xuhui Central Hospital, Fudan University (XH); and Shengli Clinical Medical College of Fujian Medical University (FJ). The study protocol was approved by the Institutional Review Board (IRB) of Zhongshan Hospital, Fudan University (Approval No. B2025-007R). For Shanghai Xuhui Central Hospital and Shengli Clinical Medical College of Fujian Medical University, formal ethical review was waived by their respective IRBs, as the study involved the retrospective analysis of fully anonymized and de-identified data in accordance with local regulatory guidelines. Informed consent was waived for this retrospective analysis of anonymized data. This study was performed in line with the principles of the Declaration of Helsinki. To ensure the representativeness of the dataset, broad inclusion criteria were applied to include all patients undergoing bone scintigraphy, regardless of the clinical indication. Exclusion criteria focused on data integrity; records that could not be reliably linked to medical history or those containing severely corrupted text were removed. The study was conducted in two phases. In Phase 1, we benchmarked four LLMs to identify the optimal model for structured information extraction. Phase 2 involved deploying the top-performing model into two distinct workflows: a human-in-the-loop workflow to evaluate gains in efficiency, and a fully automated workflow designed to construct a bone metastasis atlas from a large-scale dataset. An overview of the overall workflow is presented in Fig. 1. Overview of the workflow. LLM large language model, PPV positive predictive value, TPR true positive rate. Clinical-logic-guided prompt design We developed a five-tiered information extraction template to define the target data structure. For each tier, a controlled vocabulary consisting of mutually exclusive options was established to map diverse narrative expressions into standardized descriptions3,4,17. The template includes: Tiers 1–3: Anatomical structure description (from gross location to specific detail); Tier 4: Imaging finding description; and Tier 5: Diagnostic conclusion. The prompt engineering process was designed to enforce these clinical constraints through the following modules: 1. Role and context definition: we explicitly defined the LLM’s role, task background, and specific objectives. 2. Constrained vocabulary mapping: the controlled vocabulary for each tier was formatted as a multiple-choice list, constraining the LLM to perform a “cloze-style” selection task rather than open-ended generation. 3. Iterative constraints module: we incorporated specific instructions to address errors identified during iterative testing. This module encoded human clinical reasoning into the prompt, guiding the LLM on how to handle missing information in original narratives and how to interpret rare terminologies. 4. Workflow definition: the operational workflow for the LLM was explicitly outlined. 5. Few-Shot Examples: real-world narratives paired with expert-verified structured outputs were included as demonstrations. The final prompt was selected following a four-week iterative testing period. Example prompts and the controlled vocabulary are provided in eAppendix 1 and eAppendix 2 in Supplement 1. LLM benchmarking and model selection A benchmark dataset comprising 147 reports (49 per center) was randomly selected and annotated to serve as the ground truth. To simulate the natural epidemiological distribution of real-world clinical reports, we determined the total sample cohort size for each center during the study period, and performed simple random sampling without replacement using the pandas.sample function in Python. We evaluated four LLMs: (1) DeepSeek-V3 (version 1.0.0; DeepSeek). (2) DeepSeek-R1 (version 1.0.0; DeepSeek). (3) GPT-o3 (version 0416, OpenAI). (4) Gemini 2.5 Pro, (version Stable, Google Deepmind). All models were accessed via official application programming interfaces (APIs) with the temperature parameter set to 0 to ensure deterministic outputs. Before API processing, all narratives were strictly de-identified by assigning a unique identification number, and only the isolated narrative text was used. An example of the narrative-to-structured report conversion and the scoring methodology is depicted in Fig. 2. Workflow for narrative report structuring and quality scoring. To evaluate model performance, each structured output was compared item-by-item against an expert-curated ground truth. This reference standard was established through collaborative deliberation and consensus between two nuclear medicine specialists with 30 and 7 years of clinical experience, respectively. Metrics included the tier-based score, global quality score, true positive rate (TPR), positive predictive value (PPV), and F1-score. We analyzed performance variations for each model across the three centers and compared the performance of the different models within each specific center. The model demonstrating the highest stability and accuracy in this phase was selected for the subsequent downstream tasks. Further details are provided in eAppendix 3. Analysis of LLM outputs scoring between 0 and 5 was conducted, with errors categorized into four types: anatomical localization errors, imaging finding description errors, diagnostic conclusion errors, and composite errors. These were further stratified into three severity levels based on their potential impact on clinical decision-making: Minor errors, where LLM used synonymous expressions not strictly adhering to the controlled vocabulary without altering the final clinical conclusion; Moderate errors, where outputs contained partially inaccurate information unlikely to cause critical misjudgment (e.g., a clinically defensive statement such as “trauma possible, metastasis not excluded” being simplified by the LLM to “no definitive diagnosis”); and Major errors (factual errors), involving substantial deviation from textual facts (e.g., misclassifying benign as malignant), unwarranted hallucinated additions, or omission of pivotal clinical information. Evaluation of LLM-assisted structured reporting efficiency Thirty reports were randomly selected (ten from each center). Two nuclear medicine trainees (Trainee 1, year 3; Trainee 2, year 2) independently performed structured extraction under two conditions: (1) Unaided: manual extraction using the template alone; (2) LLM-Aided: reviewing and editing an initial output from the top-performing benchmark LLM. Completion time and structured results were recorded for each report under both conditions, with a minimum 2-week washout between sessions to reduce memory effects. Scoring followed the same criteria as the LLM benchmarking. Bone metastasis atlas construction The top-performing LLM processed all clinical narratives within the bone metastasis atlas dataset to generate structured bone scintigraphy reports. The bone metastasis status extracted by the LLM was derived directly from the diagnostic conclusions in the original narratives. To identify the primary origin of metastases, patient medical history was structurally extracted using the LLM, which were formulated by referring physicians integrating comprehensive clinical information, multimodal imaging, and longitudinal follow-up. Specifically, the LLM was instructed to analyze the free-text medical history to confirm the presence of a malignancy, extract the specific primary tumor type, and flag any patients with a history of multiple primary malignancies. Patients with multiple primary malignancies, as well as those with ambiguous or discordant clinical histories, were excluded due to the ambiguity in definitively attributing the metastatic origin. The accuracy of the patient medical history extraction was validated on a randomly selected subset of 100 cases. Leveraging the structured bone scintigraphy data, we quantified the anatomical distribution of metastatic lesions stratified by primary tumor type. An atlas was constructed focusing on the eight most common malignancies18. Statistical analysis Statistical analyses were performed using SPSS Statistics for Windows, version 27.0 (IBM Corp.) and Python, version 3.11.9 (Python Software Foundation). The performance of the LLMs and the two physicians was recorded as tier scores and global quality scores, expressed as mean and 95% confidence interval (CI). The 95% CIs were calculated based on the bootstrap method with 1000 iterations. The 95% CIs for the true positive rate and positive predictive value were calculated using the Wilson score interval. The 95% CIs for the F1 score were derived using the bootstrap method with 1000 iterations. The Kruskal-Wallis test was used to compare the performance of a single model across the three centers, and the Friedman test was used to compare the four models within each center. When overall tests were significant, post hoc pairwise comparisons with Bonferroni correction were applied. To assess the veracity of 384 lesion-based information extractions, we evaluated the categorical agreement among four LLMs using Fleiss’ kappa statistics. Concurrently, UpSet plots were utilized to visualize the distribution and intersection of extraction failures, enabling the identification of both model-specific and shared error patterns. The global quality score achieved by the two physicians with and without LLM assistance were analyzed using a mixed-design analysis of variance. A two-sided P <.05 was considered statistically significant. Results Dataset characteristics The study cohort comprised a total of 18,331 patients underwent bone scintigraphy across three medical centers. From this dataset, specific subsets were curated for model benchmarking and efficiency evaluation. For the LLM Benchmarking dataset, a randomized subset of 147 reports yielded a ground truth of 384 lesions with manually annotated 5-tier structured descriptions (ZS: 107 [27.9%]; XH: 101 [26.3%]; FJ: 176 [45.8%]). For the AI-assisted efficiency evaluation, a separate validation set (N = 30) was constructed, comprising 82 lesions (ZS: 21 [25.6%]; XH: 40 [48.8%]; FJ: 21 [25.6%]). Finally, applying the validated DeepSeek-R1 model to the entire cohort generated a comprehensive bone metastasis atlas, containing 41,151 structured lesion records that encompassed both metastatic and non-metastatic findings (Fig. 3). Database characteristics and prevalence of eight malignancies and bone metastases. a–c, Patient-based report counts (dots) and lesion-based ground truth counts (bars). d, Number of patients with malignancy history (bars) and confirmed bone metastasis (dots). LLM performance benchmarking The performance of the same model across data from different centers was compared (Table 1). DeepSeek-R1 demonstrated the best performance in structured information extraction from heterogeneous data across centers (global quality score: ZS, 4.90 (95% CI, 4.79–5.00); XH, 4.91 (95% CI, 4.82–5.00); FJ, 4.98 (95% CI, 4.96–5.00), with sufficiently stable extraction efficacy, showing no statistically significant difference in global quality scores for data from different centers (P =.277). GPT-o3 also exhibited stable performance when processing heterogeneous data from different centers (P =.072). In contrast, DeepSeek-V3 and Gemini 2.5 Pro showed unstable performance when confronted with heterogeneous content from different centers, with statistically significant differences in global quality scores across the three centers (P <.001 for both). DeepSeek-V3 demonstrated relatively lower performance on data from ZS, with a score of 4.53 (95% CI, 4.29–4.78), while Gemini 2.5 Pro also exhibited comparatively lower performance on data from ZS, with a score of 4.67 (95% CI, 4.50–4.85). The detailed scores for each tier of every LLM are provided in eTable 1 in Supplement 1. Table 1Performance of Different LLMs on Heterogeneous Data Sources.ModelsMean(95%CI)Pa valueZSXHFJDeepSeek-V34.53 (4.29–4.78)4.81 (4.72–4.90)4.93 (4.86–4.99)< 0.001aDeepSeek-R14.90 (4.79–5.00)4.91 (4.82–5.00)4.98 (4.96–5.00)0.277aGPT-o34.86 (4.78–4.94)4.84 (4.75–4.93)4.92 (4.86–4.98)0.072aGemini 2.5 Pro4.67 (4.50–4.85)4.74 (4.64–4.84)4.93 (4.89–4.97)< 0.001aPb value< 0.001b0.002b0.200bLLM large language model.aDifferences in performance of the same LLM across data from different centers.bPerformance differences among various LLMs on data from the same center. Performance of Different LLMs on Heterogeneous Data Sources. LLM large language model. aDifferences in performance of the same LLM across data from different centers. bPerformance differences among various LLMs on data from the same center. The performance of different models was compared when processing data from the same medical center. For data from ZS and XH, the four LLMs exhibited varying performance, with statistically significant differences in global quality scores across models (ZS: P <.001; XH: P =.002), with DeepSeek-R1 performing the best. In contrast, for data from FJ, no significant differences were observed in the global quality scores across all models under the guidance of the prompts (P =.200). DeepSeek-R1 demonstrated the strongest performance, achieving a TPR of 0.977 (95% CI, 0.943–0.991) and a PPV of 1.000 (95% CI, 0.978–1.000) at the FJ center. Similarly, at the XH center, it attained a TPR of 0.941 (95% CI, 0.876–0.972) and a PPV of 0.969 (95% CI, 0.940–0.990). Its performance only slightly declined when processing data from the ZS center (TPR: 0.953 [95% CI, 0.895–0.980]; PPV: 1.000 [95% CI, 0.964–1.000]). Overall, the prompt strategy we designed proved effective, with the F1 scores across all models for data processing tasks at different centers ranging from 0.860 (95% CI, 0.808–0.907) to 0.989 (95% CI, 0.976–0.997). Detailed results are presented in Table 2. Table 2TPR, PPV and F1 score of LLMs.LLMSCenterMean(95%CI)TPRPPVF1 Score meanDeepseek-V3ZS0.813 (0.729–0.876)0.946 (0.879–0.977)0.874 (0.818–0.921)XH0.832 (0.747–0.892)0.988 (0.936–0.998)0.903 (0.855–0.944)FJ0.949 (0.906–0.973)0.977 (0.941–0.991)0.963 (0.940–0.982)Deepseek-R1ZS0.953 (0.895–0.980)1.000 (0.964–1.000)0.976 (0.954–0.995)XH0.941 (0.876–0.972)1.000 (0.961–1.000)0.969 (0.940–0.990)FJ0.977 (0.943–0.991)1.000 (0.978–1.000)0.989 (0.976–0.997)GPT-o3ZS0.879 (0.803–0.928)0.989 (0.943–0.998)0.931 (0.890–0.964)XH0.871 (0.792–0.923)0.989 (0.939–0.998)0.926 (0.885–0.963)FJ0.943 (0.899–0.969)1.000 (0.977–1.000)0.971 (0.952–0.987)Gemini 2.5 ProZS0.804 (0.719–0.868)0.989 (0.938–0.998)0.887 (0.837–0.930)XH0.762 (0.671–0.835)0.987 (0.931–0.998)0.860 (0.808–0.907)FJ0.971 (0.952–0.987)1.000 (0.977–1.000)0.965 (0.941–0.983) TPR, PPV and F1 score of LLMs. Analysis of errors in LLM outputs The four LLMs demonstrated varying levels of fidelity in structured entity extraction from 384 expert-curated ground-truth entities. DeepSeek-R1 exhibited the highest accuracy, with a total error rate of only 3.9% (n = 15). In contrast, Gemini 2.5 Pro showed the highest frequency of discordance, recording a 14.8% error rate (n = 57). GPT-o3 and DeepSeek-V3 demonstrated intermediate performance with error rates of 9.3% (n = 36) and 12.0% (n = 46), respectively. Stratification by error domain revealed that anatomical structure localization was the primary source of failure across all models. 52.8% of errors in GPT-o3 and 42.1% in Gemini 2.5 Pro were localized to anatomical descriptions. Regarding error severity, Major errors were most prevalent in DeepSeek-V3 (54.3% of its error pool) and GPT-o3 (55.5%). Gemini 2.5 Pro showed a distinct profile, where the majority of its discrepancies were classified as moderate errors (54.4%) (Table 3). Table 3Performance and Error Stratification of Four LLMs in Structured Extraction from Bone Scintigraphy Reports.Evaluation CategoryMetricDeepSeek-V3DeepSeek-R1GPT-o3Gemini 2.5 ProOverall PerformanceTotal GT (N)384384384384Total errors (n,%)46 (12.0%)15 (3.9%)36 (9.3%)57 (14.8%)Error DomainsAnatomical structure16 (34.8%)6 (40.0%)19 (52.8%)24 (42.1%)Imaging findings10 (21.7%)1 (6.7%)2 (5.6%)8 (14.3%)Diagnostic conclusion10 (21.7%)4 (26.7%)11 (30.5%)19 (33.3%)Composite errors10 (21.7%)4 (26.7%)4 (11.1%)6 (10.5%)SeverityMinor11 (23.9%)1 (6.7%)4 (11.1%)6 (10.5%)Moderate10 (21.7%)5 (33.3%)12 (33.4%)31 (54.4%)Major25 (54.3%)9 (60.0%)20 (55.5%)20 (35.1%) Performance and Error Stratification of Four LLMs in Structured Extraction from Bone Scintigraphy Reports. Inter-model reliability analysis yielded a cumulative Fleiss’ Kappa of 0.3023, indicating fair agreement among the four models in their extraction outputs (Fig. 4). Gemini 2.5 Pro exhibited the highest number of unique errors (n = 25) not shared by any other model, followed by DeepSeek-V3 (n = 22) and GPT-o3 (n = 11). Conversely, only 5 entities (1.3% of the total 384 GT entities) resulted in a consensus failure across all four LLMs. DeepSeek-R1 demonstrated the most robust performance, with only 3 unique errors identified. The intersection of extraction failures among four LLMs. The horizontal bars represent the total error counts for each model, while the vertical bars indicate the size of specific error intersections. The cumulative inter-model reliability was fair, with a Fleiss’ Kappa of 0.302. Given its superior stability and reasoning accuracy across heterogeneous datasets, DeepSeek-R1 was selected as the designated reasoning engine for the subsequent downstream tasks. Efficiency gains in human-in-the-loop workflow With no LLM assistance, Trainee 1 achieved a global quality score of 4.84 (95% CI, 4.75–4.93), whereas with LLM assistance, the score increased to 4.95 (95% CI, 4.90–5.00). Similarly, Trainee 2 obtained a score of 4.88 (95% CI, 4.81–4.95) without LLM support, which rose to 4.98 (95% CI, 4.94–5.00) when assisted by LLM (Fig. 5a). With LLM assistance, the time required for Trainee 1 to complete the task decreased from 80.72 min to 14.08 min, representing an 82.6% reduction in total duration. The average time per report decreased from 2.7 min to 0.47 min. For Trainee 2, the task completion time decreased from 110.48 min to 28.12 min, a 74.5% reduction in total time, with the average time per report decreasing from 3.68 min to 0.94 min (Fig. 5b). Application and performance evaluation in downstream tasks. a–c Enhancements in diagnostic efficiency and accuracy under LLM assistance. d, Left, schematic of skeletal regions; Right, atlases illustrating bone metastasis. The quality scores of both physicians improved significantly with LLM assistance (F = 14.658; P <.001), and the enhancing effect of AI assistance was similar for both physicians (F = 0.043; P =.836). Calculated per individual tier, the error rate for Trainee 1 decreased from 3.2% (13/410) to 1.0% (4/410), while the error rate for Trainee 2 decreased from 2.4% (10/410) to 0.5% (2/410), Fig. 5c. Automated mining and atlas construction By leveraging a multi-threaded parallel processing architecture, the initial dataset of 18,331 patients was structured using DeepSeek-R1 without the need for supplementary manual annotation. After excluding 4,308 cases involving patients with multiple primary malignancies or those with ambiguous and discordant clinical histories, we included a final cohort of 14,023 patients with the eight targeted malignancies for the construction of the bone metastasis atlas. The entire procedure was completed in approximately 10 h, with a cumulative API expenditure of $10.72 (USD). The accuracy of LLMs in extracting medical history was 99.0% (99 of 100). Matching the extracted medical history with structured bone scan information revealed that within the constructed dataset, thyroid cancer exhibited the lowest bone metastasis rate at only 2.1% (28/1311), while liver cancer had the highest rate at approximately 29.1% (485/1664). We constructed an anatomical distribution atlas of bone metastases for eight major malignancies, revealing distinct predilection patterns. Notably, prostate cancer exhibited the strongest pelvic tropism (probability, 48%), whereas esophageal cancer showed a marked preference for the ribs (30%) and thoracic spine (26%). The thoracic spine (probability range, 13%−29%), ribs (probability range, 12%−30%), and pelvis (probability range, 18–48%) were identified as high-risk sites for most tumor types, while metastasis probabilities to peripheral sites such as the skull, cervical spine, and lower limbs remained consistently low (lower than 10%) (Fig. 5d). Discussion Bone scintigraphy is a nuclear medicine imaging modality effective for detecting osseous metastases of malignant tumors3,17. Analyzing existing bone scan data contributes to a better understanding of metastatic patterns across various cancer types. However, the unstructured nature of its clinical reports presents a major obstacle to data analysis. Transforming unstructured text into computable, structured data is crucial for leveraging existing data resources. While utilizing LLMs has become a viable approach to achieve this, the traditional fine-tuning pathway is often hindered by prohibitive annotation costs and specialized expertise requirements, limiting its widespread adoption in clinical settings19–22. To address this, we encode clinicians’ diagnostic logic into tailored prompts, thereby providing LLMs with immediate task-specific knowledge and ensuring high output stability. Within a human-machine collaborative workflow, this strategy not only reduces physicians’ report processing time by over 70% but also enhances extraction accuracy. A bone metastasis atlas demonstrates trends consistent with established epidemiological patterns, validating the robustness of this framework in handling large-scale real-world data. Furthermore, this low-code and scalable paradigm empowers clinicians to independently conduct large-scale data analysis. Under identical prompts, different models exhibit varying stability in data processing across different centers. DeepSeek-R1 and GPT-o3 show stable performance in structured extraction from data of three medical centers, with no statistically significant differences in scores across centers (P =.277 and 0.072, respectively). However, the heterogeneity of text styles across centers affects the stability of DeepSeek-V3 and Gemini 2.5 Pro, with these two models exhibiting statistically significant differences in scores when processing data from different centers (P <.001 for both). In the consistency analysis of model outputs based on lesions, Fleiss’ Kappa was only 0.3023 (fair agreement), revealing significant preference differences among different LLMs when processing bone scintigraphy reports. UpSet plots further confirm that the majority of errors (e.g., 25 unique errors by Gemini 2.5 Pro) are model-specific rather than arising from ambiguities in the reports themselves. Only 5 errors were common to all four LLMs, accounting for merely 1.3% of the total ground truth. For the most prevalent moderate errors in Gemini 2.5 Pro, we found they mainly stem from interpretation biases of clinical “ambiguous language” (such as defensive descriptions like “cannot exclude” or “cannot rule out other possibilities”). We observed that when original reports contained only descriptive diagnoses, the LLMs tended to output “metastatic bone tumor” in the Diagnostic conclusion if the lesion description indicated increased radiotracer uptake. This may be linked to the training data distribution of the LLMs; in clinical practice, bone scintigraphy is primarily utilized for the diagnosis of bone metastasis, and most such lesions exhibit increased radiotracer uptake. Consequently, the models may have over-inferred the findings, failing to strictly follow the prompt requirement of extracting information based exclusively on the original narrative. This interpretation inconsistency also highlights the complexity of medical information processing, making the introduction of a “human-in-the-loop” mode crucial when necessary. Physicians’ final review can effectively compensate for AI’s limitations in handling diagnostic uncertainties. In this study, DeepSeek-R1 was observed to demonstrate relatively superior performance, which is inconsistent with the results of a previous study that evaluated LLMs for medical text reasoning and information extraction in English10. This discrepancy may be attributed to the fact that its base model was primarily trained on both Chinese and English corpora, offering better adaptation for Chinese reasoning tasks23. Additionally, as DeepSeek-R1 is a reasoning model, its built-in “chain-of-thought” capability may enable it to better adhere to complex clinical constraint rules, further enhancing its performance10,23. By combining these elements with a clinical-logic-guided prompting framework, we utilized cloze-style requirements to better constrain the generation of LLMs. During the iterative prompt design process, we also observed that when presented with excessively long prompts, LLMs exhibited a tendency to disregard instructions located in the middle of the text. To address this, we adopted a strategy of reiterating rule constraints across multiple sections of the prompt and grouping constraints of the same category into specific modules. These strategies resulted in a notable improvement in model performance24. Notably, the open-source model (DeepSeek-R1) performs comparably to closed-source commercial models (e.g., GPT-o3), providing support for medical data security—hospitals can deploy high-performance open-source models on local servers, ensuring data does not leave the premises, protecting patient privacy, and achieving full-sovereign clinical data governance. Furthermore, when adapting this workflow to other linguistic environments, localizing the controlled vocabulary into the standardized medical terminology of the target language and updating the few-shot examples to reflect regional reporting styles can provide clinicians with a feasible approach to perform information extraction using general-purpose LLMs. The value of this framework is ultimately reflected in substantive optimization of clinical workflows. Experimental data show that after AI intervention, two physicians’ data organization times were reduced by 74.5% and 82.6%, respectively, with a significant improvement in average scores. Furthermore, using the top-performing DeepSeek-R1-generated large-scale bone metastasis atlas, the preferences of various malignant tumors for bone metastasis were accurately characterized, such as the pelvic tendency of prostate cancer and the preference of esophageal cancer for ribs and thoracic vertebrae. This high consistency with recognized anatomical distribution patterns not only verifies the reliability of unsupervised large-scale extraction for constructing analyzable structured databases but also lays a data foundation for deeper clinical research and model development1. Given the potential challenge of imbalanced distributions across different bone metastasis sites in clinical data mining, incorporating metadata stratification and adaptive algorithm optimization alongside the clinical-logic-guided prompting strategy presented in this study may facilitate further improvements in information extraction accuracy in future research25. Although this study demonstrates the feasibility of incorporating general-purpose LLMs into the structured extraction of bone scintigraphy reports, the ethical and safety implications associated with their widespread deployment in real-world clinical settings require careful consideration. Despite the high accessibility, current general-purpose LLMs have not yet fully met the reliability standards required for medical-grade applications concerning transparency, bias control, and hallucination mitigation, raising professional and ethical concerns26,27. Furthermore, LLMs carry the inherent risk of generating factual errors in medical contexts. For the general public or non-specialist physicians who may lack the clinical expertise to independently evaluate model outputs, such misleading information could increase the risk of clinical misjudgment or inappropriate healthcare decisions. Therefore, rigorous auditing of model performance is particularly crucial in high-risk domains such as medicine28,29. Additionally, LLMs exhibit limited robustness against training data contamination; even minor data poisoning attacks can induce the models to produce fabricated information. Given that general-purpose LLMs heavily rely on publicly available web data for pre-training, implementing effective data governance and continuous dynamic monitoring post-deployment to ensure clinical safety remains an issue necessitating further investigation30. Considering the high-stakes nature of medical practice, maintaining a “human-in-the-loop” collaborative model remains indispensable at the current stage. This serves as a critical safeguard for ensuring clinical safety and clarifying accountability for potential errors. This study still has limitations. First, the validation dataset is relatively small due to the high cost of expert-level precise annotation. Consequently, while our simple random sampling strategy reflects the natural epidemiological distribution of clinical reports, it may introduce a subtle clinical spectrum bias. Future research will focus on expanding the sample size and employing stratified sampling techniques to further mitigate this potential bias and optimize the evaluation framework. Second, the current framework remains a unimodal system and cannot correct potential imaging diagnostic errors in original reports. Future work will develop better-performing models for image diagnosis based on this database. Finally, although we conducted multi-center validation, all data originated from Chinese environments, which may cause performance variations in LLMs with different training data distributions. Conclusion This study clinically validates a clinical-logic-guided prompting framework that effectively constrains general-purpose large language models to perform structured information extraction from bone scintigraphy narratives using a predefined controlled vocabulary. By integrating this approach into a human-machine collaborative workflow, we observed improvements in both the efficiency and accuracy of clinical data processing. Furthermore, the bone metastasis atlas constructed through fully automated LLM-based extraction showed consistency with prior epidemiological knowledge. This low-code, physician-adaptable framework offers a scalable solution for deploying general-purpose LLMs in clinical settings, supporting human-in-the-loop assisted reporting and large-scale retrospective epidemiological analysis. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Author contributions Material preparation, data collection, and analysis were performed by **W.T., W.C., S.C., X.H and J.Z.**. The framework design and expert annotations were conducted by **H.S.**, **S.W.**, and **W.T.**. Efficiency evaluation was performed by **Y.X.** and **L.Z.**. The first draft of the manuscript was written by **W.T.**, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. Funding the Whole System Research Program of Long-axial and Ultra-high Time Resolution PET/CT(CEIEC-2022-ZM02-0219); the Innovative Medical Device Application Demonstration Program of Shanghai Municipal Commission of Economy and Informatization (23SHS01200); National High-End Medical Equipment Popularization and Application Program (2024TGYY06). Data availability The datasets generated and/or analysed during the current study are not publicly available due to the participants only consented to the use of their data for authorized scientific research, but are available from the corresponding author on reasonable request. Declarations Competing interests The authors declare no competing interests. Ethical approval The study protocol was approved by the Institutional Review Board (IRB) of Zhongshan Hospital, Fudan University (Approval No. B2025-007R). For Shanghai Xuhui Central Hospital and Shengli Clinical Medical College of Fujian Medical University, formal ethical review was waived by their respective IRBs, as the study involved the retrospective analysis of fully anonymized and de-identified data in accordance with local regulatory guidelines. Informed consent was waived for this retrospective analysis of anonymized data. This study was performed in line with the principles of the Declaration of Helsinki. Generative AI and AI-assisted technologies in the writing process During the preparation of this work, the authors used Gemini 2.5 Pro and DeepSeek-R1 for language polishing to enhance readability. After using these tools, the authors reviewed and edited the content as necessary and take full responsibility for the content of the published article. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Contributor Information Shuo Wang, Email: shuowang@fudan.edu.cn. Hongcheng Shi, Email: shi.hongcheng@zs-hospital.sh.cn. References Associated Data Supplementary Materials Supplementary Material 1 Data Availability Statement The datasets generated and/or analysed during the current study are not publicly available due to the participants only consented to the use of their data for authorized scientific research, but are available from the corresponding author on reasonable request.