Saying the Unsaid: Revealing the Hidden Language of Multimodal Systems Through Telephone Games

Juntu Zhao, Jialing Zhang, Chongxuan Li, Dequan Wang · 2025

Read Saying the Unsaid: Revealing the Hidden Language of Multimodal Systems Through Telephone Games by Juntu Zhao, Jialing Zhang, Chongxuan Li, Dequan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10690.txt.

Introduction

Juntu Zhao, Jialing Zhang, Chongxuan Li, and Dequan Wang peer into the hidden language of multimodal AI systems by playing telephone with them. When a system compresses an image into text and another expands the text back into an image, the systems' built-in preferences shift which concepts survive and co-occur, and those shifts are measurable.

Running multi-round telephone games, the team quantifies how strongly concepts are connected inside a system's understanding. They contribute Telescope, a dataset of more than ten thousand concept pairs, as the game's database. The method can map concept connections globally, expose training-inherited biases, and find stable pathways for fragile connections. Reasoning models then help surface unexpected relationships that go beyond textual and visual similarity.

Catalog record

CREATORJuntu Zhao, Jialing Zhang, Chongxuan Li, Dequan Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10690

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Saying the Unsaid: Revealing the Hidden Language of Multimodal Systems Through Telephone Games Juntu Zhao, Jialing Zhang, Chongxuan Li, Dequan Wang Abstract Recent closed-source multimodal systems have made great advances, but their hidden language for understanding the world remains opaque because of their black-box architectures. In this paper, we use the systems’ preference bias to study their hidden language: During the process of compressing the input images (typically containing multiple concepts) into texts and then reconstructing them into images, the systems’ inherent preference bias introduces specific shifts in the outputs, disrupting the original input concept co-occurrence. We employ the multi-round ”telephone game” to strategically leverage this bias. By observing the co-occurrence frequencies of concepts in telephone games, we quantitatively investigate the concept connection strength in the understanding of multimodal systems, i.e., ”hidden language.” We also contribute Telescope, a dataset of 10,000+ concept pairs, as the database of our telephone game framework. Our telephone game is test-time scalable: By iteratively running telephone games, we can construct a global map of concept connections in multimodal systems’ understanding. Here we can identify preference bias inherited from training, assess generalization capability advancement, and discover more stable pathways for fragile concept connections. Furthermore, we use Reasoning-LLMs to uncover unexpected concept relationships that transcend textual and visual similarities, inferring how multimodal systems understand and simulate the world. This study offers a new perspective on the hidden language of multimodal systems and lays the foundation for future research on the interpretability and controllability of multimodal systems. 1 Introduction Recent multimodal systems, particularly closed-source ones ( Hurst et al. 2024 ; StepFun 2024 ; Bai et al. 2025 ) , have made significant advances, e.g., the newest GPT-4o with Image Generation ( OpenAI 2025 ) (abbreviated as GPT-4o-IG(20250325)). However, because of these systems’ closed features, closed data, and even closed architectures, we are unable to study the systems’ understanding of the world using methods based on training. Therefore, test-time methods are urgently needed. The hidden language reflects the connection strength between concepts within multimodal systems ( Chefer et al. 2023 ) , offering insight into how they understand the world. While prior training-based methods explored it via internal features ( Chen et al. 2023 ; Chefer et al. 2023 ; Ghandeharioun et al. 2024 ) , the rise of closed-source models renders such access impossible. Hence, we investigate the hidden language of multimodal systems at test time. We innovatively propose to strategically leverage the multimodal systems’ preference bias to study their hidden language at test time. Multimodal systems are trained to fit textual and visual representations of the same scenes, which typically involves multiple interrelated concepts. Sufficient training strengthens these concept connections in systems’ hidden understanding space (abbreviated as hidden space), while limited training weakens them. Therefore, imbalanced training data brings different concept connection strengths, i.e., hidden language. As illustrated in Figure 1 , during image-to-text compression, the systems prefer to discard weakly connected concepts; during text-to-image reconstruction, the systems prefer strongly connected concepts ( Zhao et al. 2024 ) , even with the latest SOTA GPT-4o-IG(20250325). These preference biases will lead to changes in input concepts, thereby disrupting their co-occurrence in the output scene. Figure 1: Example 5-round telephone games using the latest SOTA multimodal systems released on 2025.3.25. In each image reconstruction, the system prefers stronger nearby concept connections in multimodal systems’ understanding, then changing the outputs. (Extended results on this example are provided in Appendix B ), and more examples can be found in Appendix D In this paper, we innovatively propose a test-time framework based on multi-round telephone game to leverage this preference bias, a plug-and-play method involving multiple cycles of image reconstruction. As the telephone game progresses, fragile concept combinations gradually degrade, revealing their fragile connection strength in systems’ understanding. And we quantify the connection strength (i.e., hidden language) using the concept co-occurrence frequency in the telephone game. As shown in Figure 2 , a higher co-occurrence frequency indicates a stronger concept connection. This metric captures both the training bias and generalization capability: Stronger generalization enables consistent responses to similar patterns, corresponding to a uniform connection strength distribution. We also contribute Telescope, a dataset consisting of 10,000+ concept pairs derived from 150 common visual concepts, primarily covering basic spatial relations (e.g., “ A adjacent to B ”) and some complex interactions (e.g., “ A displayed on TV screen”). Leveraging the telephone game and Telescope, we propose a scalable test-time probing framework for the hidden language of multimodal systems: Each new telephone game iteration tends to reveal new concept connections, and as test-time compute scales up, we progressively build a detailed “world map” of the multimodal hidden language. In this way: (1) We uncover key terms associated with a concept in multimodal systems’ understanding, revealing the training bias (which combinations are better-trained or not) and the systems’ generalization capability; (2) By analyzing connection strengths across multiple pathways, we can identify intermediate concepts to enhance concept connections to promote the co-occurrence of discordant concepts; (3) Reasoning-LLMs help to understand how the these connection strengths interprets physical-world laws, revealing unexpected relationships beyond textual and visual similarities. Here, we summarize our contributions: • Test-time Telephone Game Framework : We innovatively propose to reveal the hidden language of multimodal systems using the framework of telephone game and the concept co-occurrence frequency at test time; • Telescope Dataset : We contribute the Telescope, a database for systematic telephone game probing on multimodal systems’ hidden language; • Test-Time Scalable Framework : We keep on creating an increasingly comprehensive hidden language world map of multimodal systems in a scalable way. Figure 2: The longevity of concepts combinations in the telephone game (i.e., their co-occurrence frequency) quantitatively reflects the concept connections in multimodal systems’ hidden space, termed the ”hidden language.” (Lighter color means the weaker connection) 2 Related Works MultiModal Systems Recent advances in multimodal intelligence systems ( Lu et al. 2019 ; Baltrušaitis et al. 2018 ; Xie et al. 2024 ; Guo et al. 2019 ; Li et al. 2023 ; Li et al. 2022 ; Tan and Bansal 2019 ) have shown great ability in processing cross-modal information. Modular pipeline frameworks and autoregressive systems represent two typical paradigms in multimodal architecture: the former leverages V-LLMs ( Hurst et al. 2024 ; Alayrac et al. 2022 ; Liu et al. 2023 ; Wu et al. 2024 ) as core components to construct complex cross-modal connections, while the latter unifies different modalities through sequential modeling within a shared hidden space ( Team 2024 ; Chern et al. 2024 ; Li et al. 2025 ) . Furthermore, in recent years, the emergence of GPT-4o ( Hurst et al. 2024 ) marks a shift toward a fully black-box system paradigm, particularly with the latest version that integrates image generation ability into GPT-4o ( OpenAI 2025 ) . However, as multimodal systems’ internal structures become more complex and black-box, making the way they understand the world and their preferences harder to interpret. Hidden Language In traditional machine learning, researchers could intuitively examine a model’s hidden language using tools like attention maps ( Vaswani et al. 2017 ) , or Principal Component Analysis(PCA) ( Hotelling 1933 ) . As deep learning and large-scale models emerge,

Read the complete text