FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings
Santosh, Bolaji, Šimon, Oldřich, Petr · 2026
Read FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings by Santosh, Bolaji, Šimon, Oldřich, Petr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18109.txt.
Introduction
Kesiraju, Yusuf, Sedláček, Plchot, and Schwarz introduce FLiP, a method that probes what sentence embeddings retain. FLiP maps embeddings back to words in a compact factored way. Trained on multilingual, multimodal, and commercial systems, it recovers over 75 percent of the original words. That beats unfactorized rivals and exposes language and modality biases in popular encoders without running task tests.
Catalog record
| CREATOR | Santosh, Bolaji, Šimon, Oldřich, Petr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23342 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18109
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings Santosh, Bolaji, Šimon, Oldřich, Petr Abstract This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover the lexical content from multilingual (LaBSE), multimodal (SONAR) and API-based (Gemini) sentence embedding spaces in several high- and mid-resource languages. We show that FLiP can recall more than 75% of lexical content from the embeddings, significantly outperforming existing non-factorized baselines. Using this as a diagnostic tool, we uncover the modality and language biases across the selected sentence encoders and provide practitioners with intrinsic insights about the encoders without relying on conventional downstream evaluation tasks. Our implementation is public https://github.com/BUTSpeechFIT/FLiP . Kesiraju Yusuf Sedláček Plchot Schwarz keywords sentence embeddings, interpretability, SONAR, keyword extraction † † address: Speech@FIT, Brno University of Technology, Czechia † † email: {kesiraju,iyusuf,isedlacek,iplchot,schwarzp}@fit.vut.cz 1 Introduction Learning semantically aligned sentence representations agnostic of the underlying language or modality (speech, text) has several applications ranging from retrieval [ 24 ] , classification [ 17 ] and building parallel datasets across language pairs [ 6 , 5 , 11 , 21 ] . Despite the limitations due to the compression into a single vector , the research and development of embeddings models (or sentence encoders) is evolving and spanning across modalities such as image-text [ 22 ] , and audio-text [ 8 ] . Interpreting such single vector embeddings has several applications including (a) getting a sense of why a certain embedding has been retrieved (b) as a diagnostic tool for researchers and engineers to build better models, (c) as a scientific pursuit to know “ what can you cram into a single $&!# vector? ” [ 4 ] . Interpreting embeddings or, in general, the parameters of deep neural networks has gained significant attention in recent years. Among these, the linear representation hypothesis [ 20 ] is one of the simplest, elegant and theoretically grounded frameworks because of the use of linear 1 1 1 interpretable models (or projections or operations) for interpreting black-box models or embeddings. While most research and findings focus on studying text-only [ 1 ] , image-text [ 3 ] or audio-text [ 25 ] embeddings and often rely on English language for interpretation, little attention has been devoted to a comprehensive analysis of the embedding space across languages and modalities (speech and text). Log-linear models have been the core of embedding based models such as word2vec [ 18 ] , paragraph vector [ 14 ] and their variants [ 12 ] , and have shown to scale well with data. We build on the concepts from prior works and apply them for interpreting semantically-aligned sentence embeddings that are supposed to be agnostic to modality and the underlying language. Under consistent experimental settings, we also compare the “linearity of the semantics” across embeddings models such as SONAR [ 7 ] , LaBSE [ 9 ] and a black-box (API-based) Gemini embedding [ 15 ] . Although we make comparisons, a clear distinction has to be made here: our methodology should be primarily seen as diagnosis tool and not as a replacement for massive benchmarks like MTEB [ 19 ] that rank models based on various downstream task abilities. Instead our comparisons are complementary to MTEB as we dive deeper into the representation abilities of the embeddings (thereby the underlying encoders) and quantify them with more than a single number. 1. We show that a well trained factorized linear projection (FLiP is sufficient to recall 75-80% of the lexical content from a well encoded sentence embeddings, demonstrating that semantic concepts are linearly represented in embedding space. 2. We use FLiP as a diagnostic tool to systematically analyze modality alignment, language alignment, and concept language effects across SONAR, LaBSE, and Gemini embedding spaces. 3. We compare FLiP with SpLiCE [ 3 ] and show that it is objectively a better tool for interpreting the embeddings. 2 Methodology We formulate the task of interpreting embeddings via a proxy task of keyword (concept) extraction via simple linear projection (LiP). First, we introduce LiP model as general framework and then present the proposed factorized variant and the cross-lingual and cross-modal training schemes. 2.1 Log-linear model The vanilla form of the model learns a single projection matrix ( \mathbf{W} ) and a bias vector ( \mathbf{b} ) that maps a sentence embedding to the space of the vocabulary. Formally, let \mathbf{t}_{n}\in\mathbb{R}^{d} be a sentence embedding extracted from a text using a pre-trained model such as SONAR. Let \mathbf{x}_{n}\in\mathbb{W}^{|\mathcal{V}|} be the bag-of-words vector of the underlying text, where each element indicates the absolute count of the number of occurrences for word indexed at position i in an ordered vocabulary set \mathcal{V} . For each sentence embeddings, we compute the probability distribution over the words \boldsymbol{\theta}_{n}\in\triangle^{|\mathcal{V}|-1} using the aforementioned linear projection, followed by \mathrm{softmax} function. \bm{\theta}_{n}=\mathrm{softmax}(\mathbf{b}+\mathbf{W}\mathbf{t}_{n}) (1) The model is optimized to maximize the regularized log-likelihood of the training data of N samples. \mathcal{L}=\sum_{n=1}^{N}\Big[\mathbf{x}_{n}^{{}^{\mkern-1.1mu\mathsf{T}}}\,\log\bm{\theta}_{n}\Big]-\mathcal{R}(\mathbf{W}) (2) where \mathcal{R}(\mathbf{W}) denotes regularization over model parameters (word embedding matrix). 2.2 Cross-modal and cross-lingual training Since existing sentence encoders are trained yield to semantically aligned sentence embeddings for speech and text across languages, we introduce the cross-modal or cross-lingual signal into the training objective. This allows our model to compensate for any variability in language or modality (speech, text) present in the embeddings. Let \mathbf{s}_{n}\in\mathbb{R}^{d} be the speech utterance-level embedding corresponding to the \mathbf{t}_{n} . Projecting speech embedding on to the word embedding matrix, and normalizing gives: \bm{\phi}_{n}=\mathrm{softmax}(\mathbf{b}+\mathbf{W}\mathbf{s}_{n}) (3) Combining this with ( 1 ) results in the following objective \mathcal{L}=\sum_{n=1}^{N}\alpha\mathbf{x}_{n}{{}^{\mkern-1.1mu\mathsf{T}}}\log(\bm{\theta}_{n})+(1-\alpha)\mathbf{x}_{n}^{{}^{\mkern-1.1mu\mathsf{T}}}\log(\bm{\phi}_{n})-\mathcal{R}(\mathbf{W}) (4) where \alpha balances text and speech contributions. The above formulation ( 4 ) applies to bilingual text-text scenario, where \mathbf{t}_{n} and \mathbf{s}_{n} can be seen as text embedding pairs from primary (the one matching the language of the vocabulary) and secondary languages respectively. 2.3 Factorized log-linear model While the vanilla model only trains the single matrix \mathbf{W} , we find it beneficial to perform factorization (potentially low-rank) of this matrix during training: \mathbf{W}=\mathbf{AB} , where \mathbf{A}\in\mathbb{R}^{|\mathcal{V}|\times r} is the word embedding matrix and \mathbf{B}\in\mathbb{R}^{r\times d} is the modality-to-latent projection. Rank r\leq d is shared across text and speech. Though even at full-rank ( r=d ) such factorization does not change the expressive power of the transformation (it remains purely linear), in practice, we observe superior experimental results compared to the non-factorized model. We recognize that this is mainly due to the implicit regularization mechanism that the factorization imposes on \mathbf{W} [ 10 ] . Additionally, making the factorization of \mathbf{W} low-rank makes the training more efficient due to the vocabulary dimension being substantially smaller compared to the original sentence embedding dimension, while still reaping the benefits of the implicit regularization, as is demonstrated in Section 4.1 . Regularization. We incorporate L_{1} regularization on \mathbf{A} to induce sparsity. We use proximal gradient descent with a soft-thresholding operator to get explicit zeros in our solution. 2.4 FLiP: Keyword extraction from embeddings Keyword extraction requires a simple linear projection. Given an unseen sentence embedding from text or speech \mathbf{u}\in\mathbb{R}^{d} , we compute the logits and select the top- k keywords: \mathbf{z}=\mathbf{b}+\mathbf{A}(\mathbf{B}\,\mathbf{u})\\ (5) We then simply map the top- k indices in \mathbf{z} to their corresponding