Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning
Tianle Zhang, Wanlong Fang, Jonathan Woo, Paridhi Latawa, Deepak A. Subramanian, Alvin Chan, [0.5em] College of Computing, Data Science, Nanyang Technological University · 2025
Read Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning by Tianle Zhang, Wanlong Fang, Jonathan Woo, Paridhi Latawa, Deepak A. Subramanian, Alvin Chan, [0.5em] College of Computing, Data Science, Nanyang Technological University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17552.txt.
Description
Tianle Zhang and five co-authors test whether language models can reason over data that is not text, with no extra training. Their 2025 arXiv paper notes that test-time tricks often call outside tools or helper models. Wiring in the output of vision or science models has usually meant costly supervised training, which blocks quick adaptation to new fields.
Their proposal, In-Context Representation Learning, swaps the text in few-shot prompts for representations from a non-text foundation model. The language model then infers over those vectors as if they were examples it had read. The team probes the idea in the molecular domain and asks three questions: how to map the vectors in, what drives the score, and where the limits lie. As a proof of concept, the study maps a training-free route into multimodal reasoning.
Catalog record
| CREATOR | Tianle Zhang, Wanlong Fang, Jonathan Woo, Paridhi Latawa, Deepak A. Subramanian, Alvin Chan, [0.5em] College of Computing, Data Science, Nanyang Technological University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 129 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17552