TRANSPORTER : Transferring Visual Semantics from VLM Manifolds
Alexandros Stergiou · 2025
Read TRANSPORTER : Transferring Visual Semantics from VLM Manifolds by Alexandros Stergiou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18359.txt.
Introduction
Alexandros Stergiou asks how video models reach their answers, and builds a way to watch the process. Vision and language models read rich scenes with ease, yet the rules inside their choices stay opaque. New text to video tools, which make clips from words with high visual truth, hint at a probe.
Stergiou sets a new task: turn a model's score list back into video. His method, TRANSPORTER, learns a match map to the model's high sense space, so score shifts steer the clip build. The clips shift as object traits, action words, and scene context shift in the text. Tests across models show the probe gives a rich, new view of what drives a model's call.
Catalog record
| CREATOR | Alexandros Stergiou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TRANSPORTER : Transferring Visual Semantics from VLM Manifolds Alexandros Stergiou Abstract How do video understanding models acquire their answers? Although current Vision Language Models (VLMs) reason over complex scenes with diverse objects, action performances, and scene dynamics, understanding and controlling their internal processes remains an open challenge. Motivated by recent advancements in text-to-video (T2V) generative models, this paper introduces a logits-to-video (L2V) task alongside a model-independent approach, TRANSPORTER , to generate videos that capture the underlying rules behind VLMs’ predictions. Given the high-visual-fidelity produced by T2V models, TRANSPORTER learns an optimal transport coupling to VLM’s high-semantic embedding spaces. In turn, logit scores define embedding directions for conditional video generation. TRANSPORTER generates videos that reflect caption changes over diverse object attributes, action adverbs, and scene context. Quantitative and qualitative evaluations across VLMs demonstrate that L2V can provide a fidelity-rich, novel direction for model interpretability that has not been previously explored. Figure 1 : Generated videos representing VLM logit modulations with TRANSPORTER . Videos corresponding to different logit predictions are obtained by coupling VLM embeddings to generative representations. Given a modulation of a VLM caption over an object, action, or scene attribute, TRANSPORTER guides the video generation process to reflect changes made in the token logit scores. Resulting embeddings are decoded into videos aligned with logit scores (bottom videos). 1 Introduction The world is full of rich visual signals. For example, a discus throw, as in Fig. 1 , can include variations in the scene dynamics, people or object attributes, and action performances. Video understanding has experienced drastic growth with detecting, predicting, captioning, grounding, and reasoning actions from encodings Stergiou and Poppe 2025 . This convergence from visual attributes to context-rich semantics has led to Vision-Language Models (VLMs) Team et al. 2025 ; Zhang et al. 2025 ; Comanici et al. 2025 ; Abdin et al. 2024 that address tasks in tandem. Despite significant progress, uncovering explanations behind model decisions has been a longstanding challenge. Existing approaches prompt Wei et al. 2022 , linearly probe Hernandez et al. 2024 , or decode hidden embeddings Pal et al. 2023 ; Chen et al. 2024 to gain text-based descriptions that are often sensitive to changes Arditi et al. 2024 , of limited length, or misrepresent internal processes Turpin et al. 2023 . To address this gap, this paper instead revisits visual explanations and presents a logits-to-video (L2V) generative task for synthesizing videos corresponding to logit distributions. As VLMs provide answers in the form of probabilities over a token dictionary, transporting logits to videos can directly and causally represent relevant insights. To model L2V, a VLM-independent approach is introduced. TRANSPORTER couples high-semantic into visually-fidelitous representations, allowing for fine-grained generation control, aligned with VLM predictions (logits). Unlike existing methods that rely on text-based explanations or visual-saliency attributions, TRANSPORTER visualizes modulations corresponding to target attributes. During training, modulations between pairs of video attributes are encoded to latent vectors. In inference, the corresponding logits divergence is used as a condition for video generation. TRANSPORTER provides a novel interactive explainability paradigm that not only verifies the alignment between learned semantics and relevant visual representations, but also explores learned object-attribute correlations. This work’s contributions are: (i) L2V, a novel controlled video generation task for VLM visual explanations. (ii) TRANSPORTER , a model to transport global representations and local geometric relationships across embedding spaces. (iii) Learnable latents to modulate videos by the target logit divergence. (iv) Empirical evaluations and ablations across settings. 2 Related works Approaches for interpreting deep models can be divided into three groups as detailed below. Attribution visualizations relate input contributions to model predictions. For image-based models, approaches have focused on the regional saliency of categories Chattopadhay et al. 2018 ; Fong and Vedaldi 2017 ; Petsiuk et al. 2018 , object parts Bau et al. 2017 , or gradients Shrikumar et al. 2017 ; Sundararajan et al. 2017 . Extensions also propagated local relevance through attention Bousselham et al. 2025 , or layer-averaged activations Chefer et al. 2021 , while other efforts visualized attributions in video classification Stergiou et al. 2019 ; Li et al. 2021 . As recent models represent vision and language in shared embedding spaces Radford et al. 2021 ; Zhai et al. 2023 , approaches have explored text-based attributions with point-wise vision-language mutual information maximization Hernandez et al. 2021 ; Bykov et al. 2023 , score-based pairwise similarities Pach et al. 2025 ; Gong et al. 2025 , and learned ensemble models You et al. 2025 . Prompt-to-image relevance Mao et al. 2023 has also been visualized through image-level object highlights Shtedritski et al. 2023 ; Moayeri et al. 2023 , CLS token decomposition Yu et al. 2024 , and cluster graphs over latent representations Fel et al. 2023b ; Kowal et al. 2024 . In contrast, TRANSPORTER focuses on visualizing VLM semantics within the challenging video domain by generating attribute-controlled videos. Representation decomposition methods visualize parts of objects and category relations Fel et al. 2023c , or decompose activations to vector directions Graziani et al. 2023 . Vision-language approaches related object semantics to feature Gandelsman et al. 2024 ; Menon and Vondrick 2023 , or learned latent Chen et al. 2023b descriptions. Another line of research studied text attribute-to-image pattern correspondence based on embedding distances Yang and Wang 2024 ; Zhang et al. 2023 and modality-specific differences Hua et al. 2025 ; Lan et al. 2024 . Prototype-based approaches Sui et al. 2025 ; Du et al. 2022 ; Wang et al. 2023 ; Xue et al. 2024 have gained traction with contrastive-learned embeddings Du et al. 2022 , semantics-caching-instance-shifting representations Sui et al. 2025 , and semantic boundary centering Wang et al. 2023 ; Xue et al. 2024 . Optimal Transport (OT) has also been used to align visual features to text semantics Chen et al. 2023a ; Zhu et al. 2024 , and cluster features based on activations Wang et al. 2025 ; Zhao et al. 2021 . This paper explores a different direction. It uses OT to couple detailed visual representations to high-semantic embeddings for video explanations. Concept-based methods aim to invert model inputs Simonyan et al. 2013 ; Nguyen et al. 2016b ; Yin et al. 2020 ; Hatamizadeh et al. 2022 . Most objectives update inputs with Activation Maximization (AM) Simonyan et al. 2013 to increase activations of neurons Nguyen et al. 2016a , class scores Nguyen et al. 2016b , gradients Hatamizadeh et al. 2022 , features Olah et al. 2020 ; Fel et al. 2023a , or language semantics Jain and Stergiou 2025 . AM has also been adapted for convolution Feichtenhofer et al. 2020 and attention Stergiou and Deligiannis 2023 video models. Due to parameter sensitivity Yin et al. 2020 ; Stergiou and Deligiannis 2023 ; Ghiasi et al. 2022 , more recent works instead trained Koh et al. 2020 or fine-tuned Laguna et al. 2024 copies of models for visually interpretable features Koh et al. 2020 , semantic-alignment Srivastava et al. 2024 ; Kulkarni et al. 2025 , or joint vision-language representations D’Orazio et al. 2025 ; Fort and Whitaker 2025 . Concept editing has been explored for T2I models Gandikota et al. 2024 ; Hertz et al. 2023 ; Baumann et al.