Interpreting Contrastive Embeddings in Specific Domains with Fuzzy Rules
Javier Fumanal-Idocin, Mohammadreza Jamalifard, Javier Andreu-Perez · 2026
Read Interpreting Contrastive Embeddings in Specific Domains with Fuzzy Rules by Javier Fumanal-Idocin, Mohammadreza Jamalifard, Javier Andreu-Perez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12227.txt.
Introduction
Fumanal-Idocin, Jamalifard, and Andreu-Perez explain contrastive embeddings in set domains with fuzzy rules. Free text fills legal files and medical records, yet machines need structured input. CLIP turns text and images into shared vectors, but domain shifts weaken its output. A fuzzy rule-based classifier maps features of interest into the CLIP space. The learned rules show which features drive each class in plain form. Trials on clinical reports and film reviews, alone and mixed, reveal both the value and the limits of the route.
Catalog record
| CREATOR | Javier Fumanal-Idocin, Mohammadreza Jamalifard, Javier Andreu-Perez |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25729 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12227
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Interpreting Contrastive Embeddings in Specific Domains with Fuzzy Rules Javier Fumanal-Idocin, Mohammadreza Jamalifard, Javier Andreu-Perez Abstract Free-style text is still one of the common ways in which data is registered in real environments, like legal procedures and medical records. Because of that, there have been significant efforts in the area of natural language processing to convert these texts into a structured format, which standard machine learning methods can then exploit. One of the most popular methods to embed text into a vectorial representation is the Contrastive Language-Image Pre-training model (CLIP), which was trained using both image and text. Although the representations computed by CLIP have been very successful in zero-show and few-shot learning problems, they still have problems when applied to a particular domain. In this work, we use a fuzzy rule-based classification system along with some standard text procedure techniques to map some of our features of interest to the space created by a CLIP model. Then, we discuss the rules and associations obtained and the importance of each feature considered. We apply this approach in two different data domains, clinical reports and film reviews, and compare the results obtained individually and when considering both. Finally, we discuss the limitations of this approach and how it could be further improved. I Introduction Unstructured data representation is nowadays one of the most relevant sources of digital information [ 17 ] . Whether it is video [ 24 ] , text [ 9 ] or audio [ 30 ] , most existing deep learning research is built with the aim of exploiting this information for all possible kinds of tasks and degrees of supervision [ 10 ] . Most of the state-of-the-art deep learning models use the attention block, which has proven to scale particularly well with large amounts of data. One of these models is the Contrastive Language-Image Pre-training Model (CLIP) [ 36 ] , which encodes both text and image in the same vectorial space. CLIP embeddings have been used for zero-shot classification and image retrieval [ 3 ] , and have been successful in encoding texts and images from different domains [ 38 ] . However, CLIP embeddings are not always satisfactory for specific-domain applications and can present well-known biases [ 1 , 12 ] . Since the target classes are specified natural language, CLIP performance can also be affected by how those classes were specified [ 36 ] . To mitigate those problems, it is necessary to further process the CLIP features or to fine-tune the model. Fine-tuning a CLIP model can be a very expensive procedure in terms of both data and computational cost, and different methods to alleviate this have been proposed in the literature [ 40 ] . Additionally, any retraining performed on the original model can cause significant performance degradation in the model [ 40 ] . Because of that, specific strategies have been developed for individual applications, such as art classification [ 8 ] and clinical records analysis [ 23 ] . One of the most common procedures performed on CLIP features is visualization using T-SNE or PCA [ 34 ] . Many other procedures have been used to understand the embeddings obtained with these models, which is useful both for Explainable AI (XAI) and to exploit this knowledge in a larger machine learning pipeline [ 18 ] . Fuzzy rules can serve as useful tools for explainable AI [ 28 ] , but they have not been used alongside CLIP embeddings for this purpose. In this paper, we propose the use of fuzzy rules for XAI, using them to understand how the general feature embeddings are constructed by the CLIP models. Then, we perform feature extraction on the original data in order to look for variables that are semantically relevant. Finally, we map the values in those variables to the embedding space. In this way, we can study how the relevant variables in the problem affect the structure of the data in the embedding space. We used data from two different data domains: reports from patients who suffered a stroke about their rehabilitation exercises and IMDB film reviews. We compare how the embeddings obtained in both data domains reflect the desired characteristics. The rest of the paper goes as follows: in Section II , we discuss some notions and literature about text processing, CLIP models and Fuzzy rule-based classification, which is important to understand the rest of the paper. In Section III , we describe the data that we used and the fuzzy inference system that we applied. In Section IV we discuss the experiments performed and the results obtained. Finally, in Section V we give our final conclusions for this work and state our future lines. II Background II-A Text processing and sentiment Analysis Text processing refers to the manipulation, analysis, and extraction of information from raw textual data. Two fundamental steps in this process are tokenization and lemmatization. Tokenization is the initial step and consists of breaking down a piece of text into smaller units called tokens, which can be words, parts of words, or punctuation signs. Lemmatization, on the other hand, is a technique used to break down a word down to its root meaning, known as the lemma. Most tasks in natural language processing are computed after these two steps are performed [ 7 ] . Sentiment analysis is a very popular way of text processing. It focuses on the polarity of a text, and it can also study specific emotions (anger, happiness, sadness, etc.) and intentions (being interested or not interested). Techniques for sentiment analysis normally use lexicons with pre-fixed values for each word. These values can be used directly or further exploited using another algorithm [ 6 , 15 ] . It is also possible to define custom features for classification [ 6 ] . Deep learning models have achieved state-of-the-art in many natural language processing tasks [ 22 ] . However, classical techniques are still used in many real-world applications, such as customer feedback analysis, social media monitoring, and brand reputation management [ 19 ] . II-B CLIP features CLIP feature embeddings were originally proposed for general zero-shot image classification problems [ 36 ] . The CLIP model can encode the textual representation of the different target classes and the image to classify into the same vectorial space. The predicted class is the one with a minimum cosine distance between the image and that vectorial space. In addition to this initial application, additional research has tested the effectiveness of contrastive embeddings in other related domains, such as image captioning [ 4 ] , image retrieval [ 3 ] and video-to-text [ 42 ] . Besides using the CLIP model as it was originally trained in [ 36 ] , it is also possible to retrain or fine tune it to work in a particular domain. This is a popular line of research because the original CLIP model performs poorly in specific or niche tasks [ 12 ] , and because retraining a foundational model from scratch is expensive and requires large amounts of data, which might not even exist for some specific applications [ 38 ] . Another possibility is to exploit the CLIP