MIRNet: Integrating Constrained Graph-Based Reasoning with Pre-training for Diagnostic Medical Imaging

Shufeng Kong, Zijie Wang, Nuan Cui, Hao Tang, Yihan Meng, Yuanyuan Wei, Feifan Chen, Yingheng Wang, Zhuo Cai, Yaonan Wang, Yulong Zhang, Yuzheng Li, Zibin Zheng, Caihua Liu, Hao Liang · 2025

Read MIRNet: Integrating Constrained Graph-Based Reasoning with Pre-training for Diagnostic Medical Imaging by Shufeng Kong, Zijie Wang, Nuan Cui, Hao Tang, Yihan Meng, Yuanyuan Wei, Feifan Chen, Yingheng Wang, Zhuo Cai, Yaonan Wang, Yulong Zhang, Yuzheng Li, Zibin Zheng, Caihua Liu, Hao Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10013.txt.

Introduction

Shufeng Kong and fourteen co-authors present MIRNet, an approach for automated reading of medical images, tested on the hard case of tongue diagnosis. Reading such images demands fine visual and semantic judgment, while labels are scarce, labels are imbalanced, and answers must respect clinical plausibility. MIRNet combines self-supervised pre-training with constrained graph-based reasoning over expert-defined label links.

A masked autoencoder first learns visual features from unlabeled images, then graph attention models correlations among labels, with clinical priors enforced through divergence and regularization losses. Asymmetric loss and boosting ensembles counter the label imbalance. The team also releases TongueAtlas-4K, four thousand expert-annotated images with twenty-two labels, the largest public tongue data set, and reports state-of-the-art test results that should transfer to other tasks.

Catalog record

CREATORShufeng Kong, Zijie Wang, Nuan Cui, Hao Tang, Yihan Meng, Yuanyuan Wei, Feifan Chen, Yingheng Wang, Zhuo Cai, Yaonan Wang, Yulong Zhang, Yuzheng Li, Zibin Zheng, Caihua Liu, Hao Liang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10013

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MIRNet: Integrating Constrained Graph-Based Reasoning with Pre-training for Diagnostic Medical Imaging Shufeng Kong, Zijie Wang, Nuan Cui, Hao Tang, Yihan Meng, Yuanyuan Wei, Feifan Chen, Yingheng Wang, Zhuo Cai, Yaonan Wang, Yulong Zhang, Yuzheng Li, Zibin Zheng, Caihua Liu, Hao Liang Abstract Automated interpretation of medical images demands robust modeling of complex visual-semantic relationships while addressing annotation scarcity, label imbalance, and clinical plausibility constraints. We introduce MIRNet (Medical Image Reasoner Network), a novel framework that integrates self-supervised pre-training with constrained graph-based reasoning. Tongue image diagnosis is a particularly challenging domain that requires fine-grained visual and semantic understanding. Our approach leverages self-supervised masked autoencoder (MAE) to learn transferable visual representations from unlabeled data; employs graph attention networks (GAT) to model label correlations through expert-defined structured graphs; enforces clinical priors via constraint-aware optimization using KL divergence and regularization losses; and mitigates imbalance using asymmetric loss (ASL) and boosting ensembles. To address annotation scarcity, we also introduce TongueAtlas-4K, a comprehensive expert-curated benchmark comprising 4,000 images annotated with 22 diagnostic labels–representing the largest public dataset in tongue analysis. Validation shows our method achieves state-of-the-art performance. While optimized for tongue diagnosis, the framework readily generalizes to broader diagnostic medical imaging tasks. Code — https://github.com/zijie8247/MIRNet Datasets — https://doi.org/10.5281/zenodo.17557646 Introduction Medical image diagnosis requires recognizing intricate visual patterns with domain knowledge, and this process demands nuanced reasoning about statistically correlated diagnostic labels and clinical priors. In tongue analysis, for instance, “pale tongue” frequently co-occurs with “white tongue coating,” yet such domain knowledge still remain underexplored in current approaches. Recent years have witnessed significant advances in automated tongue image diagnosis, yet critical limitations persist. 7 proposed separate Residual Networks (ResNets) for individual tongue labels with late-stage output fusion, neglecting inter-label dependencies. While their work utilized a substantial dataset of 8,676 expert-annotated images covering seven categories (fissured, tooth-marked, stasis, spotted, greasy coating, peeled coating, rotten coating), this represents only a partial diagnostic spectrum, and the dataset remains non-public. 9 introduced LGAN, combining streamlined Convolutional Neural Networks (CNNs) with dual attention mechanisms (channel-wise + spatial) for feature extraction, followed by bidirectional Recurrent Neural Networks (RNNs) to model label correlations. Most recently, 8 developed IF-RCNet, a two-tier architecture employing dilated convolutions for expanded receptive fields and residual convolutional block attention modules for feature fusion, enabling segmentation-classification synergy. Despite these innovations, no existing framework systematically addresses the following interconnected challenges: (1) annotation scarcity hindering supervised learning in specialized domains, (2) severe label imbalance skewing performance toward prevalent conditions, (3) inadequate label correlation modeling limiting diagnostic coherence, and (4) unconstrained predictions generating clinically implausible outcomes. In scientific domains including biomedicine, integrating learning with reasoning has become increasingly vital — enhancing data efficiency, improving pattern recognition, and yielding scientifically valid outcomes. Exemplifying this synergy: Deep Reasoning Networks (DRNets) merge deep learning with constraint optimization to embed thermodynamic priors for automated material discovery ( 1 ) , enabling accurate phase mapping of crystal mixtures with limited unlabeled data; Physics-Informed Neural Networks (PINNs) encode governing differential equations directly into neural architectures ( 3 ) , penalizing PDE violations during training to achieve robust solutions in data-scarce engineering and biophysical applications. Yet in diagnostic medical imaging, particularly tongue analysis, this paradigm of integrating domain knowledge with data-driven learning remains critically under explored, leaving substantial potential for improved diagnostic coherence untapped. To bridge this gap, we propose MIRNet, an end-to-end architecture to integrate constrained graph-based reasoning with pre-training for tongue image diagnosis. First, our MAE Visual Encoder ( 5 ) adapts Vision Transformers to medical domains through self-supervised reconstruction of masked anatomical regions, learning transferable representations from unlabeled data. Second, the Constrained GAT Decoder ( 10 ) processes expert-defined label graphs—where nodes represent diagnostic labels and edges encode statistical co-occurrences, while enforcing clinical plausibility through a custom loss term encodes domain knowledge such as physiological incompatibilities (e.g., “thin tongue” excludes “enlarged tongue”). Third, joint optimization with asymmetric loss handles label imbalance by down-weighting negative gradients for prevalent classes. Fourth, boosting ensembles iteratively refine predictions to enhance robustness. We evaluate MIRNet on tongue image diagnosis using TongueAtlas-4K, a comprehensive benchmark curated by medical experts. This dataset contains 4,000 images annotated with 22 clinically validated labels spanning tongue color, tongue shape, property of tongue coating, and color of tongue coating. Our work delivers four key contributions to tongue image diagnosis and medical AI: • MIRNet : A pioneering framework that integrates self-supervised visual pre-training with constrained graph reasoning , addressing annotation scarcity while modeling diagnostic dependencies through clinical knowledge graphs. • TongueAtlas-4K : The largest publicly available expert-curated benchmark for tongue analysis, featuring 4,000 images annotated with 22 clinically validated labels spanning color, texture, and morphology to accelerate community research. • Differentiable clinical constraint engine : A novel constraint-aware optimization engine using KL-divergence and domain-driven regularization losses to encode medical knowledge (e.g., physiological incompatibilities) as soft constraints and thus reduce implausible predictions. The overall system further employs asymmetric loss (ASL) to mitigate label imbalance. • State-of-the-art performance : Our model consistently outperforms all baselines across multiple metrics, improving Macro Recall by 77.8% and Macro-F1 by 33.2% over the strongest competing method. Ablation studies further validate the effectiveness of our proposed components. Preliminaries This section formalizes the setting of multi‑label medical image diagnosis and highlights its characteristics. Problem Formulation Let \mathcal{X}=\mathbb{R}^{H\times W\times C} denote the medical image space and \mathcal{Y}=\{0,1\}^{K} the label space for K distinct diagnoses. Given an image \mathbf{X}\in\mathcal{X} , we aim to learn a mapping f_{\theta}:\mathcal{X}\to\mathcal{Y} that predicts a label vector \mathbf{y}=(y_{1},\dots,y_{K})^{\top} satisfying clinical knowledge constraints while accounting for data distribution characteristics: 1. Clinical Constraints : The prediction must satisfy a set of domain knowledge rules \Phi=\{\phi_{j}\}_{j=1}^{m} where \phi_{j}:\mathcal{Y}\to\{0,1\} is defined as: \displaystyle\phi_{j}(\mathbf{y})=0\quad \displaystyle\iff\quad\text{constraint }j\text{ is satisfied} \displaystyle\phi_{j}(\mathbf{y})=1\quad \displaystyle\iff\quad\text{constraint violation} with representative constraints: • Mutual exclusion : \phi_{j}(\mathbf{y})=y_{a}\cdot y_{b} (e.g., diagnoses a and b cannot co-occur) • Co-appearance : \phi_{j}(\mathbf{y})=|y_{a}-y_{b}| (e.g., a and b must both be present or both absent) • Implication : \phi_{j}(\mathbf{y})=y_{a}\cdot(1-y_{b}) (e.g., a requires presence of b ) 2. Label Imbalance : The data exhibits significant class imbalance where \exists k\in\{1,\dots,K\} such that \mathbb{P}(y_{k}=1)\leq\tau with \tau\ll 0.5 . 3. Label Dependencies : Diagnoses exhibit statistical dependencies characterized by non-zero off-diagonal covariance \Sigma_{ij}=\text{Cov}(y_{i},y_{j})\neq 0 for some i\neq j . A model for the problem must simultaneously: • Guarantee clinical plausibility: \phi_{j}(f_{\theta}(\mathbf{X}))=0\quad\forall j • Maintain robustness under class imbalance ( \tau\ll 0.5 ) • Exploit statistical dependencies ( \Sigma\neq\mathbf{I}_{K} ) Therefore, for effective tongue image diagnosis, we seek to develop a model f_{\theta} that simultaneously: (i) Addresses the data annotation scarcity, (ii) Enforces strict adherence to clinical constraints \Phi through constraint-aware optimization, (iii) Maintains robustness under severe class imbalance ( \tau\ll 0.5 ), and (iv) Exploits statistical label dependencies ( \Sigma\neq\mathbf{I}K ). This requires integrated utilization of clinical priors (via \Phi ) and statistical priors (label distributions) during training, while ensuring all predictions satisfy \phi_{j}(f\theta(\mathbf{X}))=0 \forall j\in\{1,\dots,m\} . Figure 1: The overall architecture of MIRNet. The central diagram shows the main workflow: a pretrained MAE extracts image embeddings, a label graph is built from statistical dependencies, and a domain-aware GAT captures higher-order label correlations. The model is then trained via a constraint-aware optimization mechanism. The left panel details domain-aware pretraining and the GAT, while the right panel illustrates label graph construction and constraint-aware optimization. Methodology In

Read the complete text