Benchmark for Assessing Olfactory Perception of Large Language Models
Eftychia Makri, Nikolaos Nakis, Laura Sisson, Gigi Minsky, Leandros Tassiulas, Vahid Satarifard, Nicholas A. Christakis · 2026
Read Benchmark for Assessing Olfactory Perception of Large Language Models by Eftychia Makri, Nikolaos Nakis, Laura Sisson, Gigi Minsky, Leandros Tassiulas, Vahid Satarifard, Nicholas A. Christakis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00002.txt.
Introduction
Makri and team present the Olfactory Perception benchmark, testing how well large language models reason about smell. It holds 1,010 questions across eight categories, from odor classification to mixture similarity and smell identification from real-world sources. Each question appears in two formats, compound names and isomeric SMILES. Across 21 configurations, compound-name prompts beat isomeric SMILES prompts, suggesting models access olfactory knowledge mainly through lexical associations rather than structure-based thought. The best model reaches 64.4 percent overall accuracy, showing both emerging capability and large remaining gaps.
Catalog record
| CREATOR | Eftychia Makri, Nikolaos Nakis, Laura Sisson, Gigi Minsky, Leandros Tassiulas, Vahid Satarifard, Nicholas A. Christakis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87269 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00002
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmark for Assessing Olfactory Perception of Large Language Models Thanks: Code and data: https://github.com/Satarifard/Olfactory-Perception-benchmark . Eftychia Makri, Nikolaos Nakis, Laura Sisson, Gigi Minsky, Leandros Tassiulas, Vahid Satarifard, Nicholas A. Christakis Abstract Here we introduce the Olfactory Perception (OP) benchmark, designed to assess the capability of large language models (LLMs) to reason about smell. The benchmark contains 1,010 questions across eight task categories spanning odor classification, odor primary descriptor identification, intensity and pleasantness judgments, multi-descriptor prediction, mixture similarity, olfactory receptor activation, and smell identification from real-world odor sources. Each question is presented in two prompt formats, compound names and isomeric SMILES, to evaluate the effect of molecular representations. Evaluating 21 model configurations across major model families, we find that compound-name prompts consistently outperform isomeric SMILES, with gains ranging from +2.4 to +18.9 percentage points (mean ≈ \approx +7 points), suggesting current LLMs access olfactory knowledge primarily through lexical associations rather than structural molecular reasoning. The best-performing model reaches 64.4% overall accuracy, which highlights both emerging capabilities and substantial remaining gaps in olfactory reasoning. We further evaluate a subset of the OP across 21 languages and find that aggregating predictions across languages improves olfactory prediction, with AUROC = 0.86 for the best performing language ensemble model. LLMs should be able to handle olfactory and not just visual or aural information. 1 Introduction The sense of smell occupies a unique position among human perceptual modalities ( 30 ; 43 ) . Unlike vision, where wavelengths map predictably to colors ( 46 ) , or audition, where frequencies correspond to pitch ( 38 ) , olfaction operates through a complex interplay between molecular structure and receptor biology that remains incompletely understood ( 30 ; 7 ; 8 ) . Humans can distinguish over a trillion distinct odors 8 ; 12 , though estimates are still debated ( 17 ) , yet predicting how a molecule will smell from its chemical structure alone has long eluded computational approaches ( 23 ) . This structure-odor relationship represents one of the most challenging frontiers in sensory science 23 ; 28 ; 43 . Large language models have demonstrated remarkable capabilities across diverse domains, from mathematical reasoning to code generation 6 ; 9 . Recent work has shown that these models, particularly those trained with explicit reasoning objectives ( 20 ) , can perform sophisticated tasks in chemistry, interpreting SMILES (Simplified Molecular Input Line Entry System) ( 48 ) strings, predicting reaction products, and even elucidating molecular structures from NMR spectra 41 ; 37 . These advances raise a natural question: can LLMs bridge the gap between molecular structure and sensory perception? Specifically, do these models possess knowledge about how molecules smell? This question carries both scientific and practical significance. From a scientific perspective, understanding what LLMs have learned about olfaction provides insight into how chemical knowledge is encoded in these systems. From an applied standpoint, LLMs capable of olfactory reasoning could accelerate fragrance design 34 , flavor development 15 , and the identification of malodorous contaminants in consumer products 5 . Yet despite growing interest in evaluating LLM sensory alignment across modalities including color and auditory perception, as well as taste 35 , olfaction has remained notably absent from systematic evaluation. Prior work exploring the intersection of AI and olfaction has taken different approaches. SNIFF AI 52 investigated human-AI perceptual alignment through user studies where participants described scents and embedding models attempted identification, revealing limited alignment, with only a 27.5% success rates. Other studies have examined whether LLMs can recover olfactory-semantic relationships from natural language 25 , finding that models like GPT-4o capture similarity judgments derived from word-based assessments. Meanwhile, specialized machine learning systems using graph neural networks have achieved human-level odor prediction from molecular structure 23 ; 28 . However, no existing work systematically evaluates whether general-purpose LLMs possess factual knowledge about olfactory properties through structured question-answering with objective ground-truth answers. We address this gap by introducing the Olfactory Perception (OP) benchmark, a comprehensive evaluation framework comprising 1,010 questions across eight task categories. Our benchmark spans the full complexity of olfactory perception: from basic odor detection 36 and classification to nuanced judgments of intensity and pleasantness 23 , from single-molecule descriptor identification 22 to the perceptual similarity of complex mixtures 8 ; 45 ; 39 ; 42 , and from semantic odor labeling 28 to the biological mechanisms of receptor activation 27 . Ground-truth answers are derived from established datasets in olfactory science, ensuring that our evaluation reflects genuine knowledge rather than superficial pattern matching. Odor language varies substantially across cultures and linguistic communities. Some languages support more abstract, dedicated olfactory vocabularies and enable more efficient odor naming 32 ; 31 . Motivated by this cross-linguistic diversity and evidence that human olfaction sits at the intersection of language, culture, and biology 33 , we evaluate a subset of our benchmark across multiple languages to test whether LLM olfactory knowledge is robust to linguistic framing rather than specific to English. We additionally report a cross-lingual majority-vote setting, since different languages may provide more informative descriptor inventories for different odors. A distinctive feature of our approach is the dual-prompting strategy employed throughout the benchmark. Each question is presented in two forms: one using isomeric SMILES molecular notation and another using common compound names. Because odor perception is stereospecific, different isomers, especially enantiomers, can smell very different; for example, (R)-carvone is perceived as spearmint-like whereas (S)-carvone is perceived as caraway-like. This design enables direct comparison of how different molecular representations affect olfactory reasoning, an experimental dimension unexplored in prior work. Similar methodology has proven informative in chemistry benchmarks, where canonical versus randomized SMILES representations yield different model performance 41 . The contrast between these conditions assess whether models are genuinely reasoning about molecular properties or merely retrieving associations from training data. Our evaluation encompasses both reasoning and non-reasoning models across multiple providers, including OpenAI’s GPT and o3 series, Google’s Gemini, Anthropic’s Claude, Meta’s Llama, xAI’s Grok, and DeepSeek. We systematically vary reasoning budgets where applicable, enabling analysis of how extended deliberation affects olfactory task performance, an approach that has revealed substantial performance differences in chemistry tasks 41 ; 37 . Our findings reveal that while current models achieve moderate success on certain tasks, substantial gaps remain, particularly for questions requiring genuine molecular reasoning rather than factual recall about well-known compounds. The contributions of this work are threefold. First, we introduce a comprehensive benchmark for evaluating LLMs on olfactory reasoning, comprising 1,010 questions across eight task categories grounded in peer-reviewed olfactory science. Second, we establish baseline performance across state-of-the-art models (spanning multiple providers and reasoning configurations), identifying both emerging capabilities and systematic failures. Third, via a dual-prompting methodology (compound names vs. isomeric SMILES), we provide new insight into how molecular representation format shapes model performance on perceptual prediction tasks, helping to distinguish structural reasoning from lexical association. Together, these contributions lay the groundwork for future research at the intersection of artificial intelligence and olfactory science. 2 Related work LLM benchmarks for Chemistry and Molecular Understanding. Recent benchmarks have evaluated the chemical reasoning capabilities of large language models. ChemBench 37 established a comprehensive chemistry benchmark with over 2,700 questions across eight sub-categories including organic chemistry, toxicity, and medicinal chemistry, finding that