Speak to a Protein: An Interactive Multimodal Co-Scientist for Protein Analysis
Carles Navarro, Mariona Torrens, Philipp Thölke, Stefan Doerr, Gianni De Fabritiis · 2025
Read Speak to a Protein: An Interactive Multimodal Co-Scientist for Protein Analysis by Carles Navarro, Mariona Torrens, Philipp Thölke, Stefan Doerr, Gianni De Fabritiis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17826.txt.
Introduction
Carles Navarro and four co-authors introduce a conversational helper for studying proteins. Understanding one protein usually takes weeks of reading and cross-checking. Researchers compare crystal structures, predicted structures, and bound drug-like molecules. That work is slow and needs specialized computing skills.
Their system, Speak to a Protein, answers questions in interactive dialogue. It gathers literature, structures, and ligand data behind each answer. Answers stay grounded in a live three-dimensional scene the user can inspect. The system can highlight parts, run code, and explain results with graphics. Tests covered binding pockets, shape changes, and structure-activity questions.
Catalog record
| CREATOR | Carles Navarro, Mariona Torrens, Philipp Thölke, Stefan Doerr, Gianni De Fabritiis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17826
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Speak to a Protein: An Interactive Multimodal Co-Scientist for Protein Analysis Carles Navarro, Mariona Torrens, Philipp Thölke, Stefan Doerr, Gianni De Fabritiis Abstract Building a working mental model of a protein typically requires weeks of reading, cross-referencing crystal and predicted structures, and inspecting ligand complexes, an effort that is slow, unevenly accessible, and often requires specialized computational skills. We introduce Speak to a Protein , a new capability that turns protein analysis into an interactive, multimodal dialogue with an expert co-scientist. The AI system retrieves and synthesizes relevant literature, structures, and ligand data; grounds answers in a live 3D scene; and can highlight, annotate, manipulate and see the visualization. It also generates and runs code when needed, explaining results in both text and graphics. We demonstrate these capabilities on relevant proteins, posing questions about binding pockets, conformational changes, or structure-activity relationships to test ideas in real-time. Speak to a Protein reduces the time from question to evidence, lowers the barrier to advanced structural analysis, and enables hypothesis generation by tightly coupling language, code, and 3D structures. Speak to a Protein is freely accessible at https://open.playmolecule.org . Keywords Agentic co-scientist, scientific discovery, molecular visualization, deep research, drug discovery, retrieval-augmented generation 1 Introduction Proteins are the molecular machinery of life, and understanding their structure and function is fundamental to modern biology and medicine. For a researcher in drug discovery or molecular biology, developing an intuitive, ”working mental model” of a target protein, its active sites, its conformational dynamics, and its network of interactions is a critical first step. However, this process is slow and arduous, often requiring a combination of deep domain knowledge and specialized computational skills. A researcher investigating a protein kinase to understand how a new series of inhibitors might bind must embark on a fragmented and technically demanding workflow. This involves sifting through the PubMed literature ( Sayers et al. 2020 ) , fetching and comparing multiple structures from the Protein Data Bank (PDB) ( Burley et al. 2018 ) , querying UniProt ( The UniProt Consortium 2022 ) for functional annotations and disease-associated variants, and extracting structure-activity relationship (SAR) data from databases like ChEMBL ( Mendez et al. 2018 ) . Each step requires navigating different interfaces and data formats. Furthermore, deeper analysis, such as superimposing structures, identifying key interactions, or plotting bioactivity data, requires proficiency with specialized software or scripting languages, creating a significant barrier to entry for many bench scientists. This high friction for asking and answering questions restrains curiosity and slows the pace of discovery. To address these challenges, we introduce Speak to a Protein , a new capability designed to transform protein analysis into an interactive dialogue with an AI co-scientist that collaborates with the user in real-time. Using recent advances in large language models (LLMs) ( Achiam et al. 2023 ) , our system can comprehend complex, natural language queries about a protein of interest. It autonomously retrieves, integrates, and synthesizes information from a comprehensive suite of biological data sources, including literature, structural repositories, and biochemical databases. The core innovation of Speak to a Protein is its ability to ground its responses across multiple, synchronized modalities. When a user asks a question, the AI does not simply return text. It interacts with a live 3D structural viewer to highlight residues, measure distances, or annotate binding pockets. It can generate and execute Python code in a sandboxed environment to perform calculations, filter tabular data, or generate plots on the fly. Furthermore, the AI co-scientist sees, understands and controls the visualization, offering a natural interaction with the user. This tight coupling of natural language, 3D visualization, and code execution creates a seamless and intuitive environment for scientific exploration. This paper makes the following contributions: We present the design and architecture of an AI system that integrates language, code execution, and 3D visualization for interactive protein analysis. We show how this multimodal approach drastically lowers the barrier to complex structural and biochemical data analysis. Through case studies on relevant proteins, we show that this system enables a more fluid and powerful form of hypothesis generation, accelerating the cycle from question to evidence. 2 Related Work The ambition to create an AI capable of scientific discovery is a long-standing goal, articulated in visions such as Hiroaki Kitano’s proposal for an ”AI Scientist”: a system that could autonomously formulate hypotheses, design experiments, and achieve Nobel-class discoveries ( Kitano 2021 ) . While such a fully autonomous system ( Boiko et al. 2023 ; Zou et al. 2025 ) remains a grand challenge, recent progress in large language models (LLMs) has enabled the development of a more immediate and collaborative paradigm: the ”AI co-scientist” or ”advanced intelligence” in Kitano’s words. Early systems explored conversational interfaces for structural inspection and Q&A over proteins. Guo et al. 2023 demonstrated ProteinChat , which couples LLM prompting with protein 3D structures to answer user questions about residues and pockets. Contemporary efforts such as Wang et al. 2024 and Xiao et al. 2024 investigate protein-aware prompting and multimodal conditioning for function/property reasoning. Most recently, Wang et al. 2025 proposes Prot2Chat , an LLM that fuses protein sequence, structure, and text via an early-fusion adapter, directly targeting protein Q&A. Beyond text-only chat, domain copilots increasingly drive molecular viewers and modeling tools. Sun et al. 2024 introduce ChatMol Copilot , an LLM agent that coordinates cheminformatics and modeling tools (e.g., docking, conformer generation) in response to natural-language requests. In parallel, Ille et al. 2024 systematically evaluates GPT-4’s ability to perform rudimentary structural modeling and protein–ligand interaction analysis, highlighting both promise and limitations. Our work is aligned with this line but centers on tightly coupling language, code execution, and a live 3D scene for grounded, manipulable answers. Agent frameworks augment LLMs with tool use, retrieval, and planning. ChemCrow ( Bran et al. 2024 ) shows that equipping GPT-4 with chemistry tools enables multi-step synthesis planning and materials tasks. More recently, CLADD ( Lee et al. 2025 ) proposes a retrieval-augmented multi-agent system specialized for drug discovery tasks. Speak to a Protein adopts the agentic paradigm for structural biology: it retrieves literature/structures, executes analyses (e.g., pocket mapping, SAR tables), and grounds responses in synchronized 3D visualizations. Compared to prior work, our contribution is an end-to-end, interactive co-scientist for proteins that (i) unifies literature/structure/ligand retrieval, (ii) reasons with tabular and 3D modalities, (iii) executes code for on-the-fly analyses, and (iv) directly annotates/manipulates the 3D scene in response to dialogue. This tightly coupled language–code–3D loop reduces the time from question to evidence relative to agent-only or text-only systems. Figure 1: Overview of the system architecture. The system consists of a frontend with a protein viewer and chat interface, which includes a virtual file system and Python sandbox for automated code execution and viewer manipulation. The LLM Agent is the main orchestrator that interacts with the user and calls a set of custom tools through Model Context Protocol (MCP). These tools include a literature search for a given protein and text query, retrieving specialized data through APIs such as UniProt, PDBe and ChEMBL, as well as executing Python code in a sandbox environment with a dedicated virtual file system. 3 System Overview: Speak to a Protein 3.1 Architecture Figure 2: Interactive