Synthesizing Visual Concepts as Vision-Language Programs
Antonia Wüst, Wolfgang Stammer, Hikaru Shindo, Lukas Helff, Devendra Singh Dhami, Kristian Kersting · 2025
Read Synthesizing Visual Concepts as Vision-Language Programs by Antonia Wüst, Wolfgang Stammer, Hikaru Shindo, Lukas Helff, Devendra Singh Dhami, Kristian Kersting (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18964.txt.
Introduction
Antonia Wüst and five co-authors turn visual concepts into small programs that a vision language model helps write. Such models ace broad multimodal tasks yet fail at systematic visual reasoning, with outputs that clash or break logic. Neuro-symbolic tools promise order, but their perception parts tend to be rigid and domain bound.
Their Vision-Language Programs keep the model's flexible perception and move reasoning out. The model drafts structured visual descriptions, which compile into programs that run on the image itself. Programs stay consistent with task rules and yield human readable explanations, which ease shortcut fixes. Tests on synthetic and real sets beat direct model calls. Interpretable steps are the gain.
Catalog record
| CREATOR | Antonia Wüst, Wolfgang Stammer, Hikaru Shindo, Lukas Helff, Devendra Singh Dhami, Kristian Kersting |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18964
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Synthesizing Visual Concepts as Vision-Language Programs Antonia Wüst, Wolfgang Stammer, Hikaru Shindo, Lukas Helff, Devendra Singh Dhami, Kristian Kersting Abstract Vision-Language models (VLMs) achieve strong performance on multimodal tasks but often fail at systematic visual reasoning tasks, leading to inconsistent or illogical outputs. Neuro-symbolic methods promise to address this by inducing interpretable logical rules, though they exploit rigid, domain-specific perception modules. We propose Vision-Language Programs (VLP), which combine the perceptual flexibility of VLMs with systematic reasoning of program synthesis. Rather than embedding reasoning inside the VLM, VLP leverages the model to produce structured visual descriptions that are compiled into neuro-symbolic programs. The resulting programs execute directly on images, remain consistent with task constraints, and provide human-interpretable explanations that enable easy shortcut mitigation. Experiments on synthetic and real-world datasets demonstrate that VLPs outperform direct and structured prompting, particularly on tasks requiring complex logical reasoning. 1 1 1 Project page: ml-research.github.io/vision-language-programs 1 Introduction Figure 1 : VLMs cannot reliably perform inductive logic learning from images , failing to capture visual compositions like “candles and birthday cake” . Vision-Language Programs (VLP) employ explicit symbolic reasoning to overcome such visual reasoning errors while maintaining perceptual flexibility. Vision–language models (VLMs) have achieved impressive results across multimodal tasks, yet they continue to struggle with visual reasoning. Studies reveal frequent failures in both perception and reasoning, even on relatively simple tasks ( Wüst et al. 2025 ; Kamath et al. 2023 ; Rahmanzadehgervi et al. 2024 ; Zhou et al. 2023 ; Zhang et al. 2024 ; Wang et al. 2024b ; Sarch et al. 2025 ; Helff et al. 2024 ) . E.g . , in inductive visual reasoning tasks where models must propose rules that distinguish between image sets ( cf . Fig. 1 ), VLMs often fail by generating statements that violate the task constraints. In this example, the VLM proposes the rule ”contains candle or candles”, incorrectly satisfying one of the negative images. Such errors highlight the gap between pattern recognition and systematic reasoning in VLMs. Recent work attempts to address this gap through test-time scaling, where models “think” longer via extended chain-of-thought generation ( Wei et al. 2022 ; Snell et al. 2024 ) . While effective in some cases, this approach is computationally expensive and prone to contradictions or repetitive loops ( Shojaee et al. 2025 ; Zeng et al. 2025 ; Wu 2025 ; Helff et al. 2025 ) . This raises the question of whether language-based inference alone is sufficient for robust reasoning. Neuro-symbolic (NeSy) AI offers a promising alternative by integrating neural processing with structured symbolic inference ( Garcez et al. 2015 ; Sarker et al. 2021 ; Kautz 2022 ; Manhaeve et al. 2018 ; Shindo et al. 2023 ; Shindo et al. 2024 ; Wüst et al. 2024 ) . This paradigm has demonstrated improved robustness, compositional generalization, and interpretability, e.g . , over monolithic VLMs under domain shifts ( Kamali and Kordjamshidi 2025 ) . Hereby, program synthesis ( Gulwani et al. 2017 ) , which induces interpretable and logically consistent programs from examples, provides a particularly natural mechanism for implementing this integration in visual reasoning tasks. However, existing neuro-symbolic approaches for visual reasoning face critical limitations. Methods either require explicit queries to drive program generation, limiting their applicability to inductive reasoning tasks, or depend on domain-specific image encoders, preventing generalization across diverse visual domains. We therefore propose combining VLMs with program synthesis in form of Vision-Language Programs ( VLP ) to overcome these shortcomings. Importantly, instead of embedding reasoning inside the VLM, VLP produces structured visual descriptions that can be compiled into symbolic programs, decoupling perception from reasoning. VLP automatically induces symbolic rules from small sets of labeled image examples by first discovering candidate symbols through VLM-based analysis, then defining VLM functions that extract structured representations from images, which can be composed with symbolic reasoning functions in a domain-specific language. This allows the resulting programs to leverage neural perception at execution time while maintaining symbolic interpretability and logical consistency. This dualistic process allows VLP to execute directly on images. Evaluations on both synthetic and real-world datasets demonstrate that even small VLMs, when embedded in our framework, surpass direct prompting, especially on tasks requiring complex logical reasoning. This hybrid approach thus leverages the perceptual priors of VLMs while enabling symbolic reasoning, marking a crucial step toward models that not only achieve strong performance but also provide transparent, structured decision processes. To this end, we introduce: (i) Vision-Language Programs , a framework combining VLMs with program synthesis to induce symbolic rules from labeled images without hand-crafted detectors or task-specific queries; (ii) a domain-specific language integrating compositional VLM perception functions with symbolic reasoning operators; (iii) a probabilistic synthesis procedure that discovers and ranks programs by accuracy and likelihood; (iv) comprehensive empirical evidence showing our approach outperforms direct prompting, particularly on logically complex tasks; and (v) analysis demonstrating how programmatic structure enables shortcut detection and mitigation through transparent decision processes. 2 Related Work Neuro-Symbolic Concept Induction. Neuro-symbolic AI ( Garcez et al. 2015 ; Sarker et al. 2021 ; Kautz 2022 ) seeks to integrate the strengths of neural representations with the structure of symbolic reasoning. A common approach is to extract structured representations from raw perceptual inputs, then perform rule learning over the resulting symbols ( Yang and Song 2020 ; Shindo et al. 2023 ; Shindo et al. 2024 ; Sunder et al. 2019 ; Wüst et al. 2024 ; Stammer et al. 2021 ) . These methods typically extract structured representations from raw inputs, then perform rule learning over the resulting symbols, for instance via inductive logic programming (ILP) ( Shindo et al. 2023 ; Shindo et al. 2024 ) or probabilistic logic frameworks ( Manhaeve et al. 2018 ; Skryagin et al. 2022 ; Skryagin et al. 2023 ) . While effective for complex synthetic scenes, their applicability to open-world settings is often limited due to reliance on pre-defined predicates and domain-specific object detectors for constructing intermediate symbolic representations. Program-based Visual Reasoning. A prominent recent approach to complex visual reasoning leverages programs for systematic image analysis. Frameworks such as VisProg ( Gupta and Kembhavi 2023 ) , ViperGPT ( Surís et al. 2023 ) , CodeVQA ( Subramanian et al. 2023 ) , and NePTune ( Kamali and Kordjamshidi 2025 ) employ foundation models to generate executable programs conditioned on visual inputs and natural language instructions or questions. In contrast, our approach ( VLP ) focuses on inducing programs from labeled visual examples in the absence of task-specific queries, thereby uncovering programmatic representations that explain the conceptual differences between them. Program synthesis ( Gulwani 2011 ) has long been successful in rule induction, with early efforts focusing on domains such as list processing and text editing ( Gulwani et al. 2017 ; Ellis et al. 2023 ) . A benefit of explicit program synthesis over LLM-prompted approaches is the guarantee of syntactically valid and executable programs, eliminating formatting errors that can hinder downstream reasoning. More recent work extends these ideas to abstract reasoning tasks, such as the Abstract Reasoning Corpus ( Chollet 2019 ) , by leveraging foundation models to discover higher-level concepts ( Wang et al. 2024a ; Barke et al. 2024 )