CLIP-KOA: Enhancing Knee Osteoarthritis Diagnosis with Multi-Modal Learning and Symmetry-Aware Loss Functions

Yejin Jeong, Donghun Lee · 2025

Read CLIP-KOA: Enhancing Knee Osteoarthritis Diagnosis with Multi-Modal Learning and Symmetry-Aware Loss Functions by Yejin Jeong, Donghun Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19443.txt.

Introduction

Yejin Jeong and Donghun Lee built an AI system that grades knee arthritis from X-rays more consistently than human readers often manage. Doctors grade the disease with the Kellgren and Lawrence scale, but grades vary between observers. Their system adapts CLIP, a model that links images with text, so it learns from both the X-ray and written descriptions at once.

Two added training rules push the model to agree with itself: one checks that a flipped copy of the same knee gets the same grade, and one enforces steady predictions. The system reached about 72 percent accuracy, the best reported for the task, beating plain CLIP by more than two points. The authors argue such consistency checks matter for fine-grained medical diagnosis.

Catalog record

CREATORYejin Jeong, Donghun Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-19443

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CLIP-KOA: Enhancing Knee Osteoarthritis Diagnosis with Multi-Modal Learning and Symmetry-Aware Loss Functions Yejin Jeong, Donghun Lee Abstract Knee osteoarthritis (KOA) is a universal chronic musculoskeletal disorders worldwide, making early diagnosis crucial. Currently, the Kellgren and Lawrence (KL) grading system is widely used to assess KOA severity. However, its high inter-observer variability and subjectivity hinder diagnostic consistency. To address these limitations, automated diagnostic techniques using deep learning have been actively explored in recent years. In this study, we propose a CLIP-based framework (CLIP-KOA) to enhance the consistency and reliability of KOA grade prediction. To achieve this, we introduce a learning approach that integrates image and text information and incorporate Symmetry Loss and Consistency Loss to ensure prediction consistency between the original and flipped images. CLIP-KOA achieves state-of-the-art accuracy of 71.86% on KOA severity prediction task, and ablation studies show that CLIP-KOA has 2.36% improvement in accuracy over the standard CLIP model due to our contribution. This study shows a novel direction for data-driven medical prediction not only to improve reliability of fine-grained diagnosis and but also to explore multimodal methods for medical image analysis. Our code is available at https://github.com/anonymized-link. 1 Introduction Osteoarthritis (OA) is the leading chronic musculoskeletal disease worldwide, affecting approximately 15% of the population, including around 43 million people in the United States [ 13 , 3 ] . Knee OA, in particular, is a leading cause of disability and is characterized by a progression from intermittent pain during weight-bearing activities to persistent, chronic pain [ 11 ] . OA is not merely a degenerative change in articular cartilage but a complex condition that affects the entire joint. Therefore, early diagnosis and tailored treatment strategies are essential for its effective prevention and management. One of the most widely used methods for diagnosing OA is the Kellgren & Lawrence (KL) grading system, a radiological osteoarthritis assessment framework proposed in 1957 [ 9 ] . It evaluates OA severity on a scale from 0 to 4 based on radiological (X-ray) images. However, the KL grading system relies heavily on the physician’s subjective judgment, leading to limitations that can result in variability depending on the physician’s experience [ 4 ] . In fact, the inter-rater reliability of the KL system has been reported to range from 0.54 to 0.79, with adjacent grades, such as KL 1 and KL 2, being particularly prone to misclassification due to their subtle differences [ 2 , 16 ] . To address these limitations, automated OA diagnosis using deep learning has been actively explored, highlighting the growing need for more objective and quantitative evaluation methods. Existing studies have employed various techniques for osteoarthritis(OA) diagnosis, with a particular focus on the KL grading system. However, this system has significant limitations due to high inter-observer variability and reliance on subjective assessment. To address these issues, automated KOA diagnosis systems based on deep learning and machine learning have been actively explored [ 18 ] . Some studies have utilized multi-class classification approaches to predict KL grades [ 6 , 14 , 8 ] , while others have applied ordinal regression techniques to account for the ordinal nature of KL grading [ 19 ] . Additionally, Medical Vision-Language Pretraining (VLP) has emerged as a key approach to mitigating medical data sparsity. It has been actively explored to enhance performance in a range of medical image analysis tasks [ 17 ] . Contrastive Language-Image Pretraining (CLIP) models [ 15 ] have demonstrated strong generalization capabilities across various downstream vision tasks. They have gained significant attention for their high classification accuracy, particularly in imbalanced or stylistically diverse datasets. Recent studies have explored the application of CLIP’s pretraining techniques to the medical imaging domain, leading to various reported use cases [ 22 ] . Notable examples include Diabetic Retinopathy analysis [ 20 ] , Histopathology image interpretation [ 7 ] , and PubMedCLIP, which incorporates Pathology, Blood, and Breast imaging data [ 5 ] , as well as BiomedCLIP [ 21 ] . Additionally, research on integrating ordinal relationships between classes in CLIP models has gained traction [ 20 , 10 ] , offering promising applications for tasks requiring quantitative assessment in medical imaging. In this study, we propose a CLIP-based framework (CLIP-KOA) for the automatic and consistent classification of Knee Osteoarthritis (KOA). Given the high inter-observer variability and subjective nature of the existing Kellgren-Lawrence (KL) grading system, we introduce a novel learning approach that integrates both image and text modalities to address these limitations. The main contributions of this work are threefold. First, unlike existing KOA grading classification models that rely solely on image data, CLIP-KOA integrates both image and textual information to enable more sophisticated KOA severity prediction. This allows the model to go beyond simple KL grade (0–4) classification by learning both the meaning and features of each grade, leading to a more reliable KOA assessment. Second, we leverage the inherent symmetry of KOA images to design Symmetry Loss and Consistency Loss, which ensure prediction consistency between the original and flipped images. These loss functions encourage the model to produce consistent KOA ratings regardless of image orientation, thereby improving generalization performance. The schematic illustration of CLIP-KOA is shown in Fig. 1 . Third, we empirically demonstrate that CLIP-KOA can predict KOA severity more precisely and consistently than existing CNN-based KOA grading models. While existing KL grading systems use only a fixed scale of 0–4, CLIP-KOA is designed to leverage textual information to capture more nuanced and subjective descriptions of KOA severity. This work suggests potential future research for overcoming the limitations of current KOA grading prediction methods with more refined multimodal image analysis approaches. Figure 1 : Symmetry Loss and Consistency Loss framework for knee osteoarthritis (KOA) severity grading. The original image I and its horizontally flipped version I^{H} are mapped to similarity matrices \mathcal{S} and \mathcal{S}^{H} through the image and text encoders. Symmetry Loss is computed using the cross-entropy loss between the similarity matrices and one-hot encoded labels. Jensen-Shannon Divergence (JSD) is used in \mathcal{L}_{consistency} to measure the difference between the distributions of \mathcal{S} and \mathcal{S}^{H} . The final loss function integrates Symmetry Loss and Consistency Loss to enhance the robustness of KOA severity classification. 2 Methods 2.1 Problem formulation A knee osteoarthritis (KOA) image I_{i} is provided as an input, with its corresponding severity grade G_{i} . Thus, the dataset can be represented as a collection of paired samples: \{I_{i},G_{i}\}\in D where D is the knee osteoarthritis dataset. The image encoder maps I_{i} to an image embedding x_{i} , defined as : \mathcal{X}=\{x_{i}\}^{N}_{i=1} where N is the total number of images. Given that there are K distinct grades, we define the set of text embeddings as: \mathcal{T}=\{t_{j}\}^{K}_{j=1} . The similarity matrix \mathcal{S} is computed from the set \mathcal{X} and \mathcal{T} using cosine similarity: \mathcal{S}=[s_{i,j}]_{N\times K} , s_{i,j}=x_{i}\cdot t_{j}^{T} . The objective is to model a mapping function f_{\theta}:I_{i}\xrightarrow{}G_{i} , which maps an input knee osteoarthritis image I_{i} to its corresponding grade G_{i} . This function is implemented as a deep neural network with learnable parameters \theta . 2.2 Loss function The key idea of Symmetry Loss and Consistency Loss is that the model should maintain consistent predictions even when the left and right sides of the knee X-ray

Read the complete text