An AI-powered tongue image model for home-based monitoring of liver fibrosis Authors: Xiao-Zhou Lu, Shuai Liu, Xin-Xin Lin, Yue Zeng, Ji-Hang Chen, Wei-Ping Ke, Jin-Feng Deng, Mei-Qing Cheng, Wei Li, Li-Da Chen, Zhen-Kun Lu, Bao-Guo Sun, Hang-Tong Hu, Wei Wang Published: 2025 ABSTRACT AbstractLiver fibrosis is a reversible precursor to cirrhosis, and early detection is key to halting disease progression. Tongue diagnosis provides a non-invasive and cost-effective insight into internal health; however, its subjectivity limits clinical reliability. We developed TongVMoe, a multi-task deep learning model trained on 2202 tongue images from 1601 patients, to detect liver fibrosis and simultaneously classify seven key tongue features. The model achieved an area under the curve (AUC) of 0.8061, outperforming State-of-the-Art methods such as DiffMIC-v2 (0.6929), HorNet (0.7018), InceptionNeXt (0.7012), LSNet (0.6971), and TransXNet (0.7062). TongVMoe also demonstrated robust recognition of tongue features, with AUCs of 0.9752 for cracks and 0.9232 for greasy coating. Among these features, petechiae emerged as a significant clinical indicator, showing a strong correlation with liver fibrosis (χ² = 19.516, P < 0.001). We further integrated the model into a WeChat mini-program and simulated remote screening, achieving an accuracy of 77.8% and a sensitivity of 86.2%. These findings suggest that the TongVMoe has the potential to serve as an interpretable and mobile-compatible tool for the early detection and monitoring of liver fibrosis, particularly in resource-limited areas. Trial registration: Chinese Clinical Trial Registry (ChiCTR2100053676, registered 27 November 2021).Subject terms: Biomarkers, Computational biology and bioinformatics, Diseases, Gastroenterology, Health care, Medical research Introduction Liver disease is the eleventh-leading cause of death globally, with most fatalities attributed to complications associated with cirrhosis and hepatocellular carcinoma1. As a precursor to cirrhosis, liver fibrosis develops from chronic hepatic injury but can regress with appropriate aetiologic treatment2. Therefore, dynamic monitoring and early detection of liver fibrosis are crucial for controlling disease progression and reducing the overall disease burden3 effectively. Although liver biopsy remains the reference standard for assessing fibrosis, its clinical utility is limited by its invasive nature and suboptimal intra- and inter-rater reliability4. Non-invasive tests (NITs), including serological tests5 and imaging examinations6, have been developed for the early diagnosis of liver fibrosis. However, these conventional diagnostic methods, with high acquisition and maintenance costs, are unsuitable for routine dynamic monitoring, particularly in primary care settings with limited technical capabilities and resources7. Tongue inspection, as one of the most essential diagnostic methods in Traditional Chinese Medicine (TCM), provides practitioners with valuable insights into a patient’s overall health8, including the state of internal organs and the severity of illnesses. However, traditional tongue diagnosis heavily relies on practitioner experience and is inherently subjective. Recently, with the advancement of artificial intelligence (AI), tongue diagnosis has undergone significant evolution. Innovative technologies have been developed to enhance the objectivity and accuracy of tongue diagnosis8, enabling it to serve as a screening tool for the early detection of diseases such as breast cancer9 and non-alcoholic fatty liver disease10. Also, specialized diagnostic instruments are now being applied in research and clinical practices10 to expand the scope of tongue diagnosis. The Tongue and Face Diagnosis Analysis-1 (TFDA-1) instrument, developed by Shanghai University of TCM, has been used for diabetes11 and other diseases12. Nevertheless, because they rely on large models and high-performance hardware, TCM diagnostic instruments are expensive and relatively bulky, requiring patients to visit specialized institutions for evaluation. Our study aims to develop an AI-powered tongue diagnosis system for home-based dynamic monitoring of liver fibrosis via mobile devices. We introduce the TongVMoe model, a multi-task interpretable framework that simultaneously identifies liver fibrosis and key tongue features, and further deploy it within a WeChat mini program to allow patients to upload tongue images and receive real-time diagnostic feedback (Fig. 1). This model was developed and validated on a prospective cohort using ultrasound elastography as the reference standard, and its diagnostic performance was rigorously benchmarked against a range of modern multi-task learning and state-of-the-art architectures. Complete methodological details regarding the study population, image preprocessing, network design, and statistical analysis are delineated in the Methods section. This approach facilitates accessible, continuous monitoring of liver fibrosis, paving the way for a practical follow-up tool suitable for everyday use at home. Workflow of the study. The study consists of two stages. The first stage involves developing the TongVMoe model, which is trained using tongue images based on a multi-task model. The second stage involves remote health monitoring simulation by embedding the TongVMoe model into the WeChat mini-program. Results The study consists of two stages: the development of the TongVMoe model and the remote health monitoring simulation. The workflow is presented in Fig. 1. Patient characteristics A total of 1601 patients who contributed 2202 tongue images were included in the AI model development phase. The training, validation, and test sets were comprised of 1280 (non-fibrosis: 927 [42.7 ± 10.7 years old], fibrosis: 353 [48.4 ± 12.2 years old]), 160 (non-fibrosis: 111 [42.0 ± 11.5 years old], fibrosis: 49 [47.0 ± 11.9 years old]), 161 (non-fibrosis: 107 [42.6 ± 11.8 years old], fibrosis: 54 [45.1 ± 12.2 years old]) patients, respectively. A total of 108 cases (from 103 individual patients) were enrolled in the remote health monitoring simulation, including 72 non-fibrosis cases and 36 fibrosis cases. Among them, 98 patients underwent the test once, whereas 5 patients completed the test twice at different time points. Table 1 summarizes the clinical characteristics of patients across different datasets. Table 1Baseline demographic and clinical characteristics of patients in the training, validation, and test setsTraining SetValidation SetTest SetAI model developmentLSM <7 kPaLSM ≥ 7 kPaLSM <7 kPaLSM ≥ 7 kPaLSM <7 kPaLSM ≥ 7 kPaNo. of images12675031486615167No. of patients9273531114910754Age (y)42.7 ± 10.748.4 ± 12.242.0 ± 11.547.0 ± 11.942.6 ± 11.845.1 ± 12.2Gender F406 (43.8%)78 (22.1%)39 (35.1%)14 (28.6%)41 (38.3%)14 (25.9%) M521 (56.2%)275 (77.9%)72 (64.9%)35 (71.4%)66 (61.7%)40 (74.1%)Cause of disease Hepatitis726 (78.3%)246 (69.7%)96 (86.5%)39 (79.6.2%)78 (72.9%)41 (75.9%) NAFL43 (4.6%)21 (5.9%)6 (5.4%)2 (4.1%)5 (4.7%)4 (7.4%) Others158 (17.0%)86 (24.4%)9 (8.1%)8 (16.3%)24 (22.4%)9 (16.7%)Values are presented as mean ± standard deviation (SD) for continuous variables and as numbers (percentages) for categorical variables.LSM liver stiffness measurement, NAFL non-alcoholic fatty liver. Baseline demographic and clinical characteristics of patients in the training, validation, and test sets Values are presented as mean ± standard deviation (SD) for continuous variables and as numbers (percentages) for categorical variables. LSM liver stiffness measurement, NAFL non-alcoholic fatty liver. Correlation between tongue features and liver fibrosis The analysis of the correlation between various tongue features assessed by TCM expert and hepatic fibrosis status revealed no statistically significant associations between tongue color (χ2 = 4.836, P = 0.089), cracks (χ2 = 0.637, P = 0.425), tongue coating color (χ2 = 01.873, P = 0.171), tooth marks (χ2 = 0.038, P = 0.845), tongue coating thickness (χ2 = 0.155, P = 0.561), or greasy coating (χ2 = 0.536, P = 0.464) with LSM. In contrast, petechiae exhibited a highly significant association with liver stiffness (χ2 = 19.516, P < 0.001), highlighting its potential value as a clinical indicator for liver fibrosis. Therefore, we conducted a subgroup analysis focusing on tongue petechiae to examine whether their presence affected the models’ predictive accuracy and to compare the performance of different models across these subgroups. Diagnostic performance of the TongVMoe model The TongVMoe model demonstrated robust performance across most tasks in the test set. For liver fibrosis prediction, the model achieved an accuracy of 77.98%, with a specificity of 87.42% and an AUC (Area under the curve) of 0.8061. In the task of tongue feature recognition, the model excelled in identifying cracks (Accuracy: 91.74%, AUC: 0.9752) and greasy coating (Accuracy: 90.37%, AUC: 0.9232) while displaying balanced performance in tongue coating color (Accuracy: 86.24%, AUC: 0.9310) and tooth marks (Accuracy: 87.16%, AUC: 0.9257). Additionally, the model achieved high specificity (90.80%) in the petechiae task, with an accuracy of 87.16% and an AUC of 0.8912. More details of the performance measurements of the test set are shown in Table 2. The receiver operating characteristic curves (ROC curves) for binary classification data, such as cracks, tongue coating color, tooth marks, greasy coating, petechiae, and LSM, are shown in Fig. 2a. For three-class classification data such as tongue color and tongue coating thickness, confusion matrices are illustrated in Fig. 2c and d (c: tongue color, d: tongue coating thickness) Table 2The diagnostic performance of the TongVMoe in predicting liver fibrosis and tongue features on the test setTongue FeatureAccuracy (%)Precision (%)Sensitivity (%)Specificity (%)AUCF1Tongue Color76.15 (70.64, 81.20)75.75 (70.33, 81.23)75.93 (70.71, 81.45)88.20 (85.49, 90.76)0.9081 (0.8924, 0.9387)0.7540 (0.6990, 0.8073)Cracks91.74 (88.07, 95.41)0.79.10 (69.23–88.24)92.98 (85.24-98.31)91.30 (86.71, 95.37)0.9752 (0.9558, 0.9905)0.8548 (0.7826–0.9138)Tongue Coating Color86.24 (81.19, 90.83)88.79 (82.30–94.12)85.83 (78.79–91.87)86.73 (79.56, 92.86)0.9310 (0.8877, 0.9655)0.8729 (0.8211–0.9160)Tooth Marks87.16 (82.57, 91.28)81.61 (72.86–89.53)85.54 (77.27–92.96)88.15 (82.48, 92.97)0.9257 (0.8889, 0.9566)0.8353 (0.7702–0.8925)Tongue Coating Thickness83.03 (77.98, 88.07)77.83 (70.40, 85.04)78.60 (71.74, 85.07)90.07 (86.81, 92.87)0.9394 (0.9135, 0.9641)0.7798 (0.7102, 0.8434)Greasy Coating90.37 (86.24, 94.04)92.02 (87.88–95.88)94.94 (91.08–98.01)78.33 (67.35, 88.46)0.9232 (0.8699, 0.9665)0.9346 (0.9049–0.9588)Petechiae87.16 (82.57, 91.28)66.67 (52.00–80.00)72.73 (58.82–85.30)90.80 (86.21, 94.83)0.8912 (0.8265, 0.9479)0.6957 (0.5778–0.8000)LSM77.98 (72.47, 83.49)66.67 (52.83–78.43)56.72 (44.30–68.12)87.42 (81.82, 92.64)0.8061 (0.7415, 0.8670)0.6129 (0.5000–0.7083)Values represent model performance metrics, including accuracy, precision, sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and F1 score, for each prediction task. Numbers are presented as percentages (%) or AUC values, with 95% confidence intervals (CI) shown in parentheses. LSM, liver stiffness measurement. The diagnostic performance of the TongVMoe in predicting liver fibrosis and tongue features on the test set Values represent model performance metrics, including accuracy, precision, sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and F1 score, for each prediction task. Numbers are presented as percentages (%) or AUC values, with 95% confidence intervals (CI) shown in parentheses. LSM, liver stiffness measurement. ROC curves and confusion matrices of the TongVMoe model and comparison models on the test set. a Receiver operating characteristic (ROC) curves of the TongVMoe model for cracks, tongue coating color, tooth marks, greasy coating, petechiae, and liver stiffness measurement (LSM) prediction tasks. b ROC curves of different deep learning models for liver fibrosis diagnosis based on tongue images. c Confusion matrix of the TongVMoe model for tongue color. d Confusion matrix of theTongVMoe model for tongue coating thickness classification. Comparison of different models for the prediction of liver fibrosis from tongue images We compared the TongVMoe with three multi-task learning approaches: Hard Parameter Sharing (HPS)13, Customized Gate Control (CGC)14, and DSelect-k15; and five backbone architectures: Diffusion-based Medical Image Classifier v2 (DiffMIC-v2)16, InceptionNeXt17, Large-Small Network (LSNet)18, TransXNet19, and HorNet20. Results showed the TongVMoe model exhibited superior diagnostic performance in diagnosing liver fibrosis from tongue images (Table 3). Specifically, its AUC was significantly higher than those of HPS (Accuracy: 69.72%, AUC: 0.6526) and CGC (Accuracy: 71.56%, AUC: 0.7375), and comparable to DSelect-k (Accuracy: 73.85%, AUC: 0.7643). Among the five backbone models, DiffMIC-v2 (Accuracy: 69.72%, AUC: 0.6929), InceptionNeXt (Accuracy: 71.10%, AUC: 0.7012), LSNet (Accuracy: 74.31%, AUC: 0.6971), TransXNet (Accuracy: 70.18%, AUC: 0.7062) and HorNet (Accuracy: 66.97%, AUC: 0.7018) all showed lower AUCs compared with the TongVMoe (p < 0.05). The ROC curves comparing these models are illustrated in Fig. 2b. Table 3Performance of different models in analyzing the liver fibrosis from tongue images on the test setDeep learning modelsAUCAccuracy (%)Precision (%)Sensitivity (%)Specificity (%)P valueaTongVMoe0.8061 (0.7405, 0.8627)77.98 (72.48, 83.03)66.67 (54.71–78.19)56.72 (45.16–69.23)87.42 (82.28, 92.16)—HPS0.6526 (0.5685, 0.7320)69.72 (63.76, 76.15)50.94 (37.25–64.15)40.30 (28.36–51.67)82.78 (76.62, 88.31)<0.05CGC0.7375 (0.6526–0.8177)71.56 (65.60–77.52)52.94 (42.49–64.39)67.16 (55.38–78.47)73.51 (66.67–0.80.26)<0.05DSelect-k0.7643 (0.6882, 0.8341)73.85 (67.89, 79.82)58.93 (45.83–71.16)49.25 (36.92–61.97)84.77 (78.57–90.13)0.382DiffMIC-v20.6929 (0.6078, 0.7775)69.72 (63.76, 75.23)50.57 (39.80–60.87)65.67 (55.00–76.72)71.52 (64.51, 78.53)<0.05HorNet0.7018 (0.6162, 0.7788)66.97 (61.01, 72.94)46.91 (36.11-57.32)56.72 (44.83–69.23)71.52 (64.37, 78.05)<0.05InceptionNeXt0.7012 (0.6182, 0.7807)71.10 (65.60, 77.06)52.78 (41.09–63.52)56.72 (44.44-69.23)77.48 (70.70, 84.11)<0.05LSNet0.6971 (0.6205, 0.7720)74.31 (68.35, 79.83)63.41 (48.64–78.38)38.81 (26.98–50.00)90.07 (85.03, 94.63)<0.05TransXNet0.7062 (0.6292, 0.7850)70.18 (64.68, 75.70)51.14 (40.48–61.80)67.16 (56.00–77.94)71.52 (64.81, 78.42)<0.05Values represent area under the receiver operating characteristic curve (AUC), accuracy, precision, sensitivity, specificity, and p values. Numbers are presented as percentages (%) or AUC values, with 95% confidence intervals (CI) in parentheses.aThe p-values were the comparison between the AUC of the TongVMoe model and the AUCs of the other deep learning models. Differences between various AUCs were compared using a DeLong test. Performance of different models in analyzing the liver fibrosis from tongue images on the test set Values represent area under the receiver operating characteristic curve (AUC), accuracy, precision, sensitivity, specificity, and p values. Numbers are presented as percentages (%) or AUC values, with 95% confidence intervals (CI) in parentheses. aThe p-values were the comparison between the AUC of the TongVMoe model and the AUCs of the other deep learning models. Differences between various AUCs were compared using a DeLong test. Subgroups (with/without petechiae) analysis of models in diagnosing liver fibrosis To further assess the impact of tongue petechiae on fibrosis prediction, we stratified the test set into a petechiae group (44 tongue images) and a non-petechiae group (174 tongue images). In the non-petechiae group, the TongVMoe achieved the highest AUC (0.7902) with an accuracy of 78.16%, significantly outperforming HPS (AUC: 0.6277), CGC (AUC: 0.6711), DiffMIC-v2 (AUC: 0.6098), HorNet (AUC: 0.6105), InceptionNeXt (AUC: 0.6182), LSNet (AUC: 0.6584), and TransXNet (AUC: 0.6265), while demonstrating comparable performance to DSelect-k (AUC: 0.6882). In contrast, in the petechiae group, the TongVMoe reached an AUC of 0.8606, with less pronounced differences compared to other models. Overall, all nine models demonstrated superior diagnostic performance in the petechiae group compared to the non-petechiae group, with AUCs increasing by 7.04–33.50 percentage points. Notably, HorNet (+33.50), InceptionNeXt (+32.95), and TransXNet (+32.34) exhibited the most significant gains, whereas the TongVMoe showed a modest improvement (+7.04). More details are provided in Table 4. Table 4Sub-groups analysis of different models in liver fibrosis diagnosisAUCAccuracy (%)Precision (%)Sensitivity (%)Specificity (%)P valueaPetechiae groupTongVMoe0.8606 (0.7233, 0.9531)77.27 (63.64, 88.64)76.92 (50.00, 100.00)58.82 (33.29, 80.96)88.89 (76.00, 100.00)—HPS0.7364 (0.5521, 0.8958)72.73 (59.09, 84.09)69.23 (42.83–92.86)52.94 (26.67–75.03)85.19 (70.82, 96.67)0.154CGC0.8475 (0.6758–0.9954)84.09 (72.73–95.45)77.78 (55.54–95.00)82.35 (63.14–100.00)85.19 (70.36–96.55)0.875DSelect-k0.9695 (0.9194, 1.0000)88.64 (79.49–97.73)92.86 (76.47–100.00)76.47 (54.99–95.00)96.30 (87.49, 100.00)0.063DiffMIC-v20.9085 (0.8020, 0.9913)88.64 (79.55, 97.73)83.33 (63.62–100.00)88.24 (68.74–100.00)88.89 (75.00, 100.00)0.511HorNet0.9455 (0.8625, 0.9929)81.82 (68.18, 93.18)73.68 (50.00–92.87)82.35 (61.11–100.00)81.48 (65.38, 95.65)0.210InceptionNeXt0.9477 (0.8595, 1.0000)90.91 (81.82, 97.73)88.24 (69.98–100.00)88.24 (71.43–100.00)92.59 (80.77, 100.00)0.212LSNet0.8170 (0.6762, 0.9389)75.00 (61.36, 88.64)65.00 (44.44–85.00)76.47 (52.93–94.44)74.07 (57.14, 89.29)0.542TransXNet0.9499 (0.8863, 0.9958)88.64 (79.55, 97.73)77.27 (57.14–94.74)100.00 (100.00–100.00)81.48 (65.22, 96.15)0.094Non-petechiae groupTongVMoe0.7902 (0.7123, 0.8616)78.16 (72.41, 84.48)63.64 (48.88, 77.80)56.00 (41.99, 69.64)87.10 (81.25, 92.48)—HPS0.6277 (0.5383, 0.7177)68.97 (62.07, 75.86)45.00 (30.30–60.98)36.00 (23.33–50.00)82.26 (75.20, 88.89)<0.05CGC0.6711 (0.6033, 0.7993)68.39 (60.92–75.29)46.27 (34.55–58.58)62.00 (48.27–76.01)70.97 (62.40–78.68)<0.05DSelect-k0.6882 (0.6000–0.7712)70.11 (63.22, 77.01)47.62 (32.65–63.16)40.00 (25.99–54.18)82.26 (75.19, 88.80)0.094DiffMIC-v20.6098 (0.5045, 0.7112)64.94 (57.47, 71.26)42.03 (30.88–53.63)58.00 (43.85–70.83)67.74 (59.37, 75.44)<0.05HorNet0.6105 (0.5057, 0.7098)63.22 (56.32, 70.11)38.71 (26.15–51.86)48.00 (33.33–62.27)69.35 (61.34, 77.78)<0.05InceptionNeXt0.6182 (0.5203, 0.7112)66.09 (59.20, 72.99)41.82 (28.57–55.36)46.00 (32.00–60.78)74.19 (66.66, 81.82)<0.05LSNet0.6584 (0.5659, 0.7466)74.14 (67.82, 80.46)61.90 (40.00–82.15)26.00 (14.89–39.13)93.55 (88.79, 97.54)<0.05TransXNet0.6265 (0.5185, 0.7214)65.52 (58.05, 72.99)42.42 (30.90–54.29)56.00 (42.37–69.82)69.35 (60.66, 77.10)<0.05Values represent area under the receiver operating characteristic curve (AUC), accuracy, precision, sensitivity, specificity, and p values. Numbers are presented as percentages (%) or AUC values, with 95% confidence intervals (CI) in parentheses.aThe P values were the comparison between the AUC of the TongVMoe model and the AUCs of the other deep learning models in the petechiae and non-petechiae groups, respectively. Differences between various AUCs were compared using a DeLong test. Sub-groups analysis of different models in liver fibrosis diagnosis Values represent area under the receiver operating characteristic curve (AUC), accuracy, precision, sensitivity, specificity, and p values. Numbers are presented as percentages (%) or AUC values, with 95% confidence intervals (CI) in parentheses. aThe P values were the comparison between the AUC of the TongVMoe model and the AUCs of the other deep learning models in the petechiae and non-petechiae groups, respectively. Differences between various AUCs were compared using a DeLong test. Interpretability of the AI diagnosis The interpretability of the TongVMoe lies in both its architecture and visualization results. Structurally, the model produced explicit predictions for each tongue feature alongside the fibrosis classification demonstrated previously, thereby revealing the underlying associations between observable features and disease states. For visual explanations of the model’s decision-making process, we applied EigenCAM21 to observe the regions attended by the TongVMoe model during prediction. We evaluated whether these areas corresponded to clinical features commonly examined by physicians. Analysis of the heatmaps revealed that the model’s highlighted regions closely matched the areas clinicians typically focused on. Figure 3a–d illustrate four representative cases where the model accurately diagnosed liver fibrosis by focusing on clinically relevant tongue features. For instance (Fig. 3a), when tongue cracks were present, the model focused on the cracked areas, consistent with the regions clinically assessed by physicians, underscoring the clinical validity of the AI model’s predictions. Visualization of attended regions generated by EigenCAM during tongue diagnosis. a Tongue image exhibiting cracks. b Tongue image with tooth marks. c Tongue image showing petechiae. d Tongue image with thick and greasy coating. Home monitoring simulation using the TongVMoe model We integrated the TongVMoe model into a WeChat mini-program that passed the hospital’s review process. Figure 4 presents the interface of the developed mini-program for tongue image acquisition and liver fibrosis assessment. Overall, the mini-program achieved an accuracy of 77.8%, a sensitivity of 86.2%, and a specificity of 73.7%. User interface of the mini-program for liver fibrosis screening based on tongue image analysis. The mini-program comprises three sections: a the homepage, b the tongue image acquisition page, and c the result page. The original Chinese text in the interface has been translated into English, with translations provided according to the numbered annotations (①–⑨): ① Tongue Image-Based Liver Fibrosis Intelligent Evaluation System ② Upload tongue image ③ Rapid detection ④ Intelligent Analysis ⑤ Start ⑥ Ensure that the entire tongue is positioned within the shooting frame; Adequate lighting, no backlight, no exposure, no reflection; Make sure the tongue is not stained ⑦ Result ⑧ Tongue Analysis: Damp-heat syndrome, Tongue color: Red; Tongue coating color: Yellow; Cracks: None; Tooth marks: Yes; Tongue coating thickness: Medium; Greasy coating: Yes; Petechiae: None. ⑨ Screening Result: Liver fibrosis (+); Referral to hepatology is recommended. Notably, among the 36 cases diagnosed with liver fibrosis, 27 were single-time measurements from individual patients, while the remaining nine cases came from five follow-up patients who underwent elastography twice at different time points. Of these five patients, four were consistently identified as having liver fibrosis in both examinations. In contrast, one patient initially tested negative but was later diagnosed with liver fibrosis during a follow-up visit three months later. Remarkably, the mini-program predicted liver fibrosis in this patient during the second visit as the disease progressed. Discussion In this study, we developed a deep-learning AI model, the TongVMoe, designed to screen for liver fibrosis and identify associated tongue features. Furthermore, we integrated the model into a WeChat mini-program and simulated a remote health monitoring scenario for liver fibrosis screening. Our study demonstrates that, with the assistance of AI, tongue images can be used not only for the preliminary screening of liver fibrosis but also as a tool for monitoring disease progression in patients with chronic liver disease during routine follow-ups. Previously, we investigated the use of tongue images to develop an AI model capable of detecting liver fibrosis22. However, the model lacked interpretability regarding relevant tongue features, which undermined the trust of both physicians and patients in the diagnostic results and limited its clinical applicability. In addition, the training dataset was relatively small, resulting in suboptimal model performance. In the current study, we addressed both issues by enhancing the model’s ability to recognize tongue features and expanding the training dataset to boost overall performance and generalizability. According to TCM theory, physicians can assess the condition of the liver by observing specific tongue features23. However, tongue diagnosis is highly subjective, and the diversity of tongue features further complicates its assessment. In this context, petechiae serve as a crucial diagnostic indicator. The subtle reddish-purple dots are considered in TCM to correspond to impaired circulation and stagnation of liver Qi and blood. From a modern perspective, this may parallel the microcirculatory disturbances that occur during the progression of hepatic fibrosis. Although visually less prominent than features such as cracks or greasy coating, petechiae provided stronger pathophysiological specificity for fibrosis. This may explain why all models exhibited better performance in the petechiae subgroup. In contrast, the non-petechiae subgroup, which presented more heterogeneous and nonspecific features, posed greater challenges for fibrosis detection. Notably, the TongVMoe model maintained stable performance across both subgroups, underscoring its generalizability in real-world clinical settings where tongue presentations are diverse. Overall, these findings reveal a gap between visual prominence and clinical relevance, suggesting that future model development should strike a balance between visually apparent features and clinically meaningful yet subtle ones. Deep learning is often considered a “black box,” but Explainable Artificial Intelligence (XAI)24 offers solutions by providing objective and interpretable insights. Heatmaps generated by EigenCAM showed that the TongVMoe model consistently focused on clinically relevant tongue regions when predicting different features, supporting its visual interpretability. In addition to such visual evidence, the multi-task framework allows the model to reason about tongue features in parallel with disease diagnosis, thereby partially simulating the holistic and pattern-based diagnostic thinking of TCM physicians. This not only enhances transparency but also bridges the gap between AI decision-making and clinical reasoning, providing a more intuitive and trustworthy interpretation for practitioners. We simulated a remote health scenario in which patients captured tongue images using their smartphones. Despite the potential decline in image quality, the AI model maintained robust performance and high sensitivity, enabling early detection of liver abnormalities. The follow-up results further support the model’s potential for early warning and longitudinal monitoring of liver fibrosis, highlighting its practical value in dynamic disease management, particularly in primary care and rural settings. Importantly, patient privacy is strictly protected, as no personal information is collected. Our study has several limitations. First, we selected ultrasound elastography as the diagnostic reference standard due to its feasibility and reliability. Although ultrasound elastography has demonstrated promising results for the non-invasive assessment of liver fibrosis25, its accuracy may be influenced by factors such as the patient’s body mass index (BMI), the degree of liver inflammation, and the type of equipment used. These potential variabilities may introduce noise into the reference labels and influence the model’s performance evaluation. Second, the tongue images for the mini-program were all captured during clinic visits. Although the pictures were taken using patients’ smartphones, the lighting conditions were relatively uniform. This setting does not fully reproduce the complex lighting and positional variations that are likely to occur in real home environments, which may affect the model’s performance in broader real-world applications. Third, the dataset for simulating home monitoring was relatively small, mainly consisting of single-time measurements, which limited the validation of the model’s longitudinal predictive ability and dynamic monitoring performance. Moreover, the study was conducted in a single clinical center, and external validation across different populations and imaging conditions is still needed to ensure generalizability. In future studies, we plan to expand the dataset and further refine the tongue diagnosis model to improve its robustness and clinical applicability. In conclusion, this study provides preliminary evidence supporting the use of deep learning-based tongue image analysis as a non-invasive approach to liver fibrosis screening. While further validation in larger and more diverse populations is required, our findings suggest that the AI model could potentially serve as an assistive tool for remote monitoring and follow-up management in chronic liver disease. Methods This prospective study was approved by the Institutional Committee on Ethics (ICE) for Clinical Research and Animal Trials at the First Affiliated Hospital of Sun Yat-sen University (Approval No. 2021464), and informed consent was obtained before collecting the tongue images from each participant. This study was registered at the Chinese Clinical Trial Registry (ChiCTR2100053676; registered on 27 November 2021) in accordance with the World Health Organization International Clinical Trials Registry Platform (WHO ICTRP) requirements. Participants Our study was conducted in two stages. Stage 1 involved the development of the AI diagnostic model. During this phase, patients who met the following criteria were enrolled between April 2021 and March 2024: (1) age ≥ 18 years; (2) a prior diagnosis of liver diseases, including chronic hepatitis, non-alcoholic fatty liver disease (NAFLD), abnormal transaminase levels and other related conditions, was confirmed in the Department of Gastroenterology at our hospital; (3) consented to tongue image collection. In Stage 2, participants were invited to test the mini-program during outpatient visits before undergoing liver elastography in the Department of Medical Ultrasonics between August 2024 and March 2025. Tongue images collection and annotation Participants were required to fast for 4–6 h before undergoing the ultrasound examination, ensuring that their tongues’ appearances were not affected by food or beverages. After receiving ultrasound elastography, patients were instructed to extend their tongues naturally for the collection of tongue images. In the first phase, photos were captured in no-flash mode using a Sony DSC-RX100 camera, and multiple images were taken for some patients to ensure data diversity. In the second phase, patients scanned a QR code to access the mini-program, then used their smartphones to capture a tongue image or upload a pre-existing one. Adhering to data privacy principles, our tongue diagnosis WeChat mini-program did not collect patient data, and diagnostic results were recorded on-site. There were no restrictions on the type of smartphone or camera resolution. We invited a senior TCM physician with 20 years of clinical experience to annotate tongue images by identifying and labeling their tongue features. To standardize the process, we categorized the features into seven main types, each further divided into two or three subcategories based on tongue features commonly observed in clinical practice. Table 5 details these tongue features and their corresponding indications in TCM theory. The physician referred to the categories outlined in this table during the annotation process to ensure consistency. Table 5Details of tongue features, their Traditional Chinese Medicine (TCM) indications, and distributions across the training, validation, and test datasetsTongue FeaturesSub-ClassTCM IndicationsTraining N* = 1770Validation N* = 214Test N* = 218Tongue ColorPale/Pale redRedDark redQi, Blood, or Yang deficiencyExcess of Heat or Yin deficiencySevere Excess of Heat or Blood Stasis687602481847654926957CracksYesNoQi/Yin deficiency, or Heat syndrome-48812826814657161Tongue Coating ColorWhiteYellowCold, Dampness, or normal healthHeat syndrome4889689711798120Tooth MarksYesNoSpleen Qi deficiency, or Dampness-67410967214283135Tongue Coating ThicknessThinMediumThickNormal or early stage of diseaseModerate state of Dampness, PhlegmSevere Dampness, Phlegm, or Heat104244927912943421316324Greasy CoatingYesNoDampness, Phlegm, or food retention-12205501318315860PetechiaeYesNoBlood Stasis or Heat in the Blood-29314774017444174Values represent the number of tongue images (N) in each dataset. N*, total number of images in the respective dataset. Details of tongue features, their Traditional Chinese Medicine (TCM) indications, and distributions across the training, validation, and test datasets Pale/Pale red Red Dark red Qi, Blood, or Yang deficiency Excess of Heat or Yin deficiency Severe Excess of Heat or Blood Stasis 687 602 481 84 76 54 92 69 57 Yes No Qi/Yin deficiency, or Heat syndrome - 488 1282 68 146 57 161 White Yellow Cold, Dampness, or normal health Heat syndrome 488 968 97 117 98 120 Yes No Spleen Qi deficiency, or Dampness - 674 1096 72 142 83 135 Thin Medium Thick Normal or early stage of disease Moderate state of Dampness, Phlegm Severe Dampness, Phlegm, or Heat 1042 449 279 129 43 42 131 63 24 Yes No Dampness, Phlegm, or food retention - 1220 550 131 83 158 60 Yes No Blood Stasis or Heat in the Blood - 293 1477 40 174 44 174 Values represent the number of tongue images (N) in each dataset. N*, total number of images in the respective dataset. Reference standard: ultrasound elastography examination Ultrasound elastography was selected as the reference standard for assessing the degree of fibrosis in this study, with radiologists having at least 10 years of experience performing real-time two-dimensional shear wave elastography (2D SWE). The patient lay supine, and the transducer was positioned in the intercostal space to obtain clear images of the liver. The propagation of shear waves was measured, and the stiffness value, often expressed in kilopascals (kPa), was calculated. The procedure lasted for 10–20 min. The cutoff value of Liver stiffness measurement (LSM) was set to 726, dividing the patients into “fibrosis (LSM ≥ 7 kPa)” and “non-fibrosis (LSM < 7 kPa)”. Image pre-processing Tongue images were divided at an 8:1:1 ratio into training, validation, and test sets. Stratified sampling was conducted based on sub-attributes, ensuring that the class distribution within each attribute remained approximately consistent across all sets (Table 5). There were no overlapping patients between data sets. We employed the TongueSAM27 to segment the tongue images and minimize interference from irrelevant content. For the input tongue images, we performed a series of data augmentation and normalization operations to simulate various real-world scenarios, thereby enhancing the model’s generalizability and robustness. The data pre-processing pipeline included the following steps: The input images were resized to 256 × 256 pixels using bilinear interpolation; Random padding was performed with a padding of 20 pixels and following with a random crop to 224 × 224 pixels; Horizontal flipping was applied with a probability of 0.5; Random affine transformations was applied including rotations (±10°) and translations (up to 5% of image dimensions); Sharpness adjustment was randomly applied with a sharpening factor of 2.0 (p = 0.3); Random transformations was performed with a randomized sequence including rotation (±10°, p = 0.5), perspective distortion (scale = 0.2, p = 0.5), sharpness adjustment (factor=2.0, p = 0.3), and Gaussian blur (kernel size 5–9, sigma 0.1–5.0, p = 0.4); Then images were converted to tensor format and normalized using dataset-specific statistics (mean = [0.765, 0.73, 0.746], standard deviation = [0.28, 0.315, 0.309]). This diverse range of augmented samples improved the model’s generalization ability. In contrast, the validation and test phases employed only basic preprocessing (resizing, tensor conversion, and normalization) to ensure consistent evaluation conditions. Network architecture Our preliminary research22 demonstrated the effectiveness of the deep learning model for tongue diagnosis in liver fibrosis. However, the model lacks interpretability, limiting its ability to provide clinically meaningful explanations. This time, we employed the state-of-the-art visual VMamba28 for feature extraction and Multi-gate Mixture-of-Experts (MMoE)29 as the network backbone. Feature Extraction Network: VMamba utilizes a 2D Selective Scan (SS2D) module to scan the image along four directions, efficiently capturing both local and global characteristics. It learns hierarchical visual representations by stacking Visual State Space (VSS) blocks, effectively serving diverse tasks such as tongue feature classification and liver fibrosis diagnosis. Multi-task Learning with MMoE: The features extracted by VMamba are fed into the MMoE module. MMoE generates independent gates for each task to control the contributions of shared experts, modeling inter-task relationships and learning task-specific feature combinations. Given input features x, the shared bottom layers learn a common representation z=fx;θ. Then, K expert networks ϕk,k=1,2,...,K extract representations softly shared across tasks. The output for thei-th task is: wheregikz is the gating network for the k-th expert and the i-th task. The final prediction is: Gradient Balancing with Conflict-Averse Gradient(CAGrad)30: CAGrad was employed to balance the gradients across the eight tasks and mitigate task imbalance. Let Liθ be the loss for task i. The average loss is: CAGrad finds an update vector d by solving: where g0=1K∑i=1Kgi is the average gradient, and c∈[0,1) is a hyperparameter. Finally, the task-specific features are input to the corresponding prediction heads for attribute classification and liver fibrosis diagnosis tasks. Figure 1 shows the schematic diagram of the model architecture. Training process The model was trained on a single NVIDIA H100 GPU with a batch size of 16 for 100 epochs. The optimizer was Adam with a learning rate of 1 × 10⁻⁵ and weight decay of 1 × 10⁻⁷. The learning rate scheduler employed a cosine annealing strategy (CosineAnnealingLR) with Tmax = 100 and ηmin = 1e-6. An early stopping mechanism was applied, halting training if the change in validation AUC over the past 10 epochs was less than 0.01. Other key hyperparameters included: Dropout Rate (0.5), Drop Path Rate (0.5), MTL CAGrad cα (0.5), and specific initialization for MLP layers. For binary classification tasks, a probability threshold of 0.5 was applied, where predictions with probabilities greater than or equal to 0.5 were classified as positive. For multi-class classification tasks, the model produced a probability distribution over all categories through a softmax layer, and the class with the highest predicted probability was assigned as the final label. Comparison with other multi-task learning models and modern architectures To comprehensively assess the robustness and generalizability of our model, we compare it with both widely used multi-task learning methods and recently developed backbone designs. For multi-task learning, we include three representative models: HPS, CGC, and DSelect-k, each adopting different strategies for feature sharing across tasks. In addition, we benchmark our approach against five state-of-the-art backbone architectures: DiffMIC-v2, InceptionNeXt, LSNet, TransXNet, and HorNet. These models encompass diverse design philosophies, including diffusion-based learning (DiffMIC-v2), modernized convolutional networks (InceptionNeXt), integrated self-attention module (LSNet), hybrid token mixing (TransXNet), and high-order interaction modeling (HorNet). The performance of each model was evaluated using the same dataset, where we analyzed their effectiveness in diagnosing liver fibrosis. Statistical analysis Continuous variables were described as mean ± standard deviations (SD) and were compared by the t-test or Mann–Whitney U test. The association between individual tongue characteristics (e.g., presence of cracks or tooth marks) and liver fibrosis status was analyzed using the Pearson χ² test. The performances of all models were evaluated in terms of Area Under Curve (AUC), accuracy (ACC), precision (Pre), sensitivity (Sen), specificity (Spe), and F1-score(F1). AUC values were compared using the DeLong method. Acc, Pre, Rec, Spe, and F1 were compared using a paired samples t-test after verifying the normality of differences. Results with two-sided P-values of less than 0.05 indicated a statistically significant difference. Confidence intervals (CIs) were computed at a level of 95% using 1000 bootstrap samples. The analyses were conducted using Python 3.8 (Python Software Foundation). Acknowledgements This study was supported by the National Nature Science Foundation of China (NO: 82371983 and NO: 82272076), the Guangxi Key Research and Development Project (No. GuikeAB25069464), and the Guangxi Science and Technology Major Special Project (No. GuikeAA23073013). Author contributions X.Z.L.: Conceptualization, Formal Analysis, Methodology, Data analysis and interpretation, Writing-Original Draft Preparation, Writing-Review & Editing; S. L.: Data Curation, Model development, Data analysis and interpretation, Writing-Original Draft Preparation; X.X.L.: Literature review, Model implementation and validation, Statistical analysis, Figure preparation, Writing-Manuscript Revision; Y.Z.: Data Curation, Methodology, Software, Writing-Original Draft Preparation; J.H.C.: Data Curation, Formal Analysis, Model Development; W.P.K.: Software, Validation, Model Development, Supervision; J.F.D.: Data Curation, Formal Analysis, Investigation, Methodology; M.Q.C.: Literature review, Statistical analysis, Writing-Manuscript Revision; W.L.: Data Curation, Methodology, Investigation; L.D.C.: Project Administration, Validation; Z.K.L.: Project Administration, Supervision, Model development, Writing-Manuscript Revision; B.G.S.: Project Administration, Supervision, Data Curation, Writing-Review & Editing; H.T.H.: Conceptualization, Model Development, Methodology, Supervision, Writing-Review & Editing; W.W.: Conceptualization, Project Administration, Supervision, Writing-Review & Editing. Data availability The tongue images and corresponding clinical metadata generated and used in this study are not publicly available due to patient privacy considerations. However, de-identified data can be made available from the corresponding author upon reasonable request and with approval from the Ethics Committee of the First Affiliated Hospital of Sun Yat-sen University. Code availability The code used for model development, training, and evaluation in this study is openly available at https://github.com/MedAI-UAIX/TongVMoe. Additional scripts related to data preprocessing and deployment can be obtained from the corresponding author upon reasonable request. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. These authors contributed equally: Xiao-Zhou Lu, Shuai Liu, Xin-Xin Lin, Yue Zeng, Ji-Hang Chen. Contributor Information Zhen-Kun Lu, Email: lzk06@sina.com. Bao-Guo Sun, Email: sunbaog@mail.sysu.edu.cn. Hang-Tong Hu, Email: huht5@mail.sysu.edu.cn. Wei Wang, Email: wangw73@mail.sysu.edu.cn. References Associated Data Data Availability Statement The tongue images and corresponding clinical metadata generated and used in this study are not publicly available due to patient privacy considerations. However, de-identified data can be made available from the corresponding author upon reasonable request and with approval from the Ethics Committee of the First Affiliated Hospital of Sun Yat-sen University. The code used for model development, training, and evaluation in this study is openly available at https://github.com/MedAI-UAIX/TongVMoe. Additional scripts related to data preprocessing and deployment can be obtained from the corresponding author upon reasonable request.