MolVision: Molecular Property Prediction with Vision Language Models
Deepan Adak, Yogesh Singh Rawat, Shruti Vyas · 2025
Read MolVision: Molecular Property Prediction with Vision Language Models by Deepan Adak, Yogesh Singh Rawat, Shruti Vyas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03283.txt.
Introduction
Adak, Rawat, and Vyas introduce in 2025 MolVision, a method that predicts molecule properties with vision language models. Property prediction drives drug and materials work. Past language model work used text codes for molecules, which can be ambiguous and hide structure.
Their method feeds both molecule images and text descriptions to the model. A benchmark spans ten datasets across classification, regression, and description tasks. Nine vision language models were tested in zero shot, few shot, and tuned settings. Images alone were not enough, but image plus text fusion improved generalization. Tuning the vision side with a light method helped further.
Catalog record
| CREATOR | Deepan Adak, Yogesh Singh Rawat, Shruti Vyas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03283
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MolVision: Molecular Property Prediction with Vision Language Models Deepan Adak, Yogesh Singh Rawat, Shruti Vyas Abstract Molecular property prediction is a fundamental task in computational chemistry with critical applications in drug discovery and materials science. While recent works have explored Large Language Models (LLMs) for this task, they primarily rely on textual molecular representations such as SMILES/SELFIES, which can be ambiguous and structurally less informative. In this work, we introduce MolVision, a novel approach that leverages Vision-Language Models (VLMs) by integrating both molecular structure as images and textual descriptions to enhance property prediction. We construct a benchmark spanning ten diverse datasets, covering classification, regression and description tasks. Evaluating nine different VLMs in zero-shot, few-shot, and fine-tuned settings, we find that visual information improves prediction performance, particularly when combined with efficient fine-tuning strategies such as LoRA. Our results reveal that while visual information alone is insufficient, multimodal fusion significantly enhances generalization across molecular properties. Adaptation of vision encoder for molecular images in conjunction with LoRA further improves the performance. The code and data is available at : https://molvision.github.io/MolVision/ . 1 Introduction Recent advancements in Large Language Models (LLMs) have revolutionized natural language understanding and generation across multiple domains [ 1 ] . Models such as GPT [ 2 , 3 ] , LLaMA [ 4 ] , and Mistral [ 5 ] have demonstrated exceptional capabilities in reasoning, knowledge retrieval, and complex problem-solving. Extending beyond pure text-based reasoning, Vision-Language Models (VLMs) integrate visual and textual modalities [ 1 , 6 , 7 , 8 ] , enabling them to perform tasks such as image captioning, visual question answering, and multimodal retrieval with remarkable success. While VLMs have been extensively explored in computer vision and NLP applications, their potential in scientific domains—particularly in chemistry—remains largely unexplored. Given that molecular structures are inherently visual, leveraging vision in molecular analysis presents an exciting, yet underexplored, research direction. Recent works such as ChemLLM [ 9 ] and ChemLLM-Bench [ 10 ] have begun to explore LLMs for molecular property prediction. These methods primarily rely on textual molecular representations, such as SMILES and SELFIES, which have been widely used in cheminformatics for decades. However, these representations have notable limitations, including their non-uniqueness and syntactic instability, where structurally identical molecules may have vastly different textual encodings. This ambiguity introduces challenges for LLMs, which process molecular structures as linear strings, potentially overlooking key structural relationships. While some approaches attempt to improve these representations through graph-based models [ 11 ] , the integration of easily available visual molecular data remains largely unexplored in this domain. Incorporating visual information has the potential to significantly enhance molecular property prediction. Chemists usually analyze molecular structures using bond-line or skeletal diagrams to infer properties such as reactivity, toxicity, and solubility. These visual representations inherently encode structural and spatial information that textual descriptors may fail to capture. For example, subtle differences in geometry, stereochemistry, or electron delocalization can have profound effects on molecular properties, yet are difficult to represent accurately in SMILES format alone. By leveraging VLMs, which are designed to process both visual and textual inputs, we aim to bridge this gap and improve predictive modeling in cheminformatics. To this end, we introduce MolVision, a multimodal benchmark for molecular property prediction. In contrast to prior works MolVision integrates both textual and visual representations (Figure 1 ). Our benchmark spans ten diverse datasets, covering classification, regression and description tasks across a wide range of molecular properties, including toxicity, solubility, and bioactivity. We evaluate nine different VLMs in zero-shot, few-shot, and fine-tuned settings, providing a comprehensive analysis of their performance in this domain (Figure ). We also propose a simple contrastive strategy to adapt visual component of VLMs for this domain and demonstrate its effectiveness for property prediction. Figure 1: MolVision comparison: Comparison of relevant molecular property prediction approaches. Through extensive experimentation, we uncover several key insights. First, while VLMs struggle in zero-shot settings, their performance improves significantly with in-context learning and fine-tuning. Second, efficient adaptation techniques such as LoRA enhance the predictive accuracy and generalization to unseen molecular properties. Third, while visual information alone is insufficient for accurate property prediction, combining molecular images with textual representations yields notable performance gains specifically for larger molecules. These findings suggest that vision-augmented molecular modeling presents a promising avenue for future research in AI-driven chemistry, with numerous potential applications. We make the following contributions: • We introduce MolVision, a novel approach for molecular property prediction, integrating molecular structure images with textual representations. • We present a multimodal benchmark and systematically assess nine state-of-the-art VLMs in zero-shot, few-shot, and fine-tuned settings across ten datasets highlighting their strengths and limitations for property prediction. • We show that efficient adaptation of VLMs for property prediction enhances both performance and generalization, and that combining visual and textual data significantly improves molecular property prediction. • We propose a simple contrastive strategy, implemented with LoRA, to efficiently adapt vision aspect of VLMs for this domain and demonstrate it effectiveness for property prediction. Figure 2: Overview of visual-textual approach for property prediction: The image representation along with textual description are used as input by the VLM where the image is encoded by a vision encoder and textual description is encoded by a text encoder. These multimodal features are used to generate the output with the help of a decoder. We show template prompt used for property prediction, including general outline, task instruction, in-context learning (ICL with k=2), and an image prompt. 2 Related works Property prediction: Prior research in molecular property prediction has explored various methods and representations [ 12 ] . Traditional approaches, like molecular fingerprints [ 13 , 14 ] and descriptors [ 15 ] , rely on expert knowledge but are limited in capturing complex data relationships. Recently, machine learning techniques [ 16 , 17 , 18 ] , particularly graph-based methods like graph convolutional networks (GCNs) [ 19 ] , have gained prominence for capturing molecular interactions. Additionally, deep learning models, including RNNs, CNNs, and transformers [ 20 , 21 , 22 , 23 , 24 ] , have shown strong performance in modeling structural and sequential information from molecular data. Multimodal foundational models: Recent advances in Foundational Models [ 5 , 4 , 25 ] have shown the ability of multimodal LLMs to process both vision and language. These models integrate vision encoders [ 26 , 27 ] with LLMs [ 28 , 4 ] for generating text responses. Models like Llama Adapter V2 [ 7 ] and Flamingo [ 29 ] explored multimodal structures. Typically, these models pre-train on image caption datasets [ 30 , 31 ] and fine-tune on task-specific datasets [ 32 ] . Models such as Llava 1.5 [ 33 ] and QwenVL [ 34 ] are designed for instruction-following tasks but may struggle with science-specific challenges like computational chemistry. Foundation models for property prediction: Recent efforts have explored LLMs for property prediction, such as ChemLLM [ 9 ] , ChemLLMBench [ 10 ] , and Nach0 [ 35 ] . ChemLLM [ 9 ] utilizes ChemData, an instruction-tuning dataset, to address the need for specialized models in chemistry. Guo et al. [ 10 ] assess LLMs in chemistry, focusing on understanding and reasoning tasks with