Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning

Tianle Zhang, Wanlong Fang, Jonathan Woo, Paridhi Latawa, Deepak A. Subramanian, Alvin Chan, [0.5em] College of Computing, Data Science, Nanyang Technological University · 2025

Read Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning by Tianle Zhang, Wanlong Fang, Jonathan Woo, Paridhi Latawa, Deepak A. Subramanian, Alvin Chan, [0.5em] College of Computing, Data Science, Nanyang Technological University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17552.txt.

Description

Tianle Zhang and five co-authors test whether language models can reason over data that is not text, with no extra training. Their 2025 arXiv paper notes that test-time tricks often call outside tools or helper models. Wiring in the output of vision or science models has usually meant costly supervised training, which blocks quick adaptation to new fields.

Their proposal, In-Context Representation Learning, swaps the text in few-shot prompts for representations from a non-text foundation model. The language model then infers over those vectors as if they were examples it had read. The team probes the idea in the molecular domain and asks three questions: how to map the vectors in, what drives the score, and where the limits lie. As a proof of concept, the study maps a training-free route into multimodal reasoning.

Catalog record

CREATORTianle Zhang, Wanlong Fang, Jonathan Woo, Paridhi Latawa, Deepak A. Subramanian, Alvin Chan, [0.5em] College of Computing, Data Science, Nanyang Technological University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE129 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-17552

Browse the full library · About the Tower of Babel