Promptception: How Sensitive Are Large Multimodal Models to Prompts?
Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, Salman Khan · 2025
Read Promptception: How Sensitive Are Large Multimodal Models to Prompts? by Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, Salman Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-03986.txt.
Introduction
A 2025 research paper by Mohamed Insaf Ismithdeen and two co-authors measures how sensitive large multimodal models are to the wording of their prompts. On multiple-choice question answering, changes in phrasing can swing accuracy by up to fifteen percent for some model and prompt combinations. That variability undermines evaluation, since reported scores often come from chosen best-case prompts.
Their framework, Promptception, systematises the test: sixty-one prompt types grouped into fifteen categories and six supercategories, each probing one aspect of prompt formulation. Ten models are evaluated across three multiple-choice benchmarks, with GPT-4o, Gemini 1.5 Pro, and open-source systems among them. Proprietary models prove more sensitive to phrasing, a sign of tighter alignment with instruction wording. Open-source models are steadier, though they struggle more with nuanced and complex phrasing. From the analysis, the authors propose prompting principles tailored separately to proprietary and open-source models.
Catalog record
| CREATOR | Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, Salman Khan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-03986
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Promptception: How Sensitive Are Large Multimodal Models to Prompts? Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, Salman Khan Abstract Despite the success of Large Multimodal Models (LMMs) in recent years, prompt design for LMMs in Multiple - Choice Question Answering (MCQA) remains poorly understood. We show that even minor variations in prompt phrasing and structure can lead to accuracy deviations of up to 15% for certain prompts and models. This variability poses a challenge for transparent and fair LMM evaluation, as models often report their best-case performance using carefully selected prompts. To address this, we introduce Promptception , a systematic framework for evaluating prompt sensitivity in LMMs. It consists of 61 prompt types, spanning 15 categories and 6 supercategories, each targeting specific aspects of prompt formulation, and is used to evaluate 10 LMMs ranging from lightweight open - source models to GPT-4o and Gemini 1.5 Pro, across 3 MCQA benchmarks: MMStar, MMMU - Pro, MVBench. Our findings reveal that proprietary models exhibit greater sensitivity to prompt phrasing, reflecting tighter alignment with instruction semantics, while open - source models are steadier but struggle with nuanced and complex phrasing. Based on this analysis, we propose Prompting Principles tailored to proprietary and open-source LMMs, enabling more robust and fair model evaluation. 1 Introduction Recent advancements in Large Multimodal Models (LMMs) have significantly improved their ability to integrate vision and language, enabling strong performance on a range of reasoning tasks involving textual and visual information Radford et al. 2021 ; OpenAI et al. 2024 ; Chen et al. 2025 . These models take visual cues (single image, multiple images, and video) and text as input, to output a textual response. They have been fine-tuned on a variety of tasks, including captioning, visual question-answering (VQA) Li et al. 2025 , visual grounding Rasheed et al. 2024 ; Munasinghe et al. 2025 , and temporal grounding Ren et al. 2024 . Figure 1: Categorization of prompts proposed in our Promptception framework. It consists of 61 prompt types, spanning 15 categories (e.g. Answer Handling, Penalty-Based Prompts, Poor Linguistic Formatting) and 6 supercategories (e.g. Task-Specific Instructions, Choice Formatting and Presentation), providing a comprehensive evaluation framework for assessing LMM sensitivity to prompts. However, despite these successes, the systematic study of sensitivity of LMMs remains an open question. While the impact of texual prompt variations has been extensively studied in Large Language Models (LLMs) Alzahrani et al. 2024b ; Balepur et al. 2024 as well as CLIP-based Vision-Language Models (VLMs) Wu et al. 2023 , its effects on LMMs have not been systematically explored. Understanding how these models respond to textual modifications is crucial for assessing their reliability in multimodal tasks. Visual Question Answering (VQA) has emerged as a fundamental evaluation framework for assessing the capabilities of LMMs Agrawal et al. 2016 . VQA tasks are typically structured as either open-ended or multiple-choice questions (MCQ) Chen et al. 2024 ; Li et al. 2024 . While open-ended responses provide flexibility, they pose challenges in evaluation due to ambiguity and the need for complex answer-matching techniques. Consequently, MCQ formats are widely adopted in recent image and video benchmarks Zhang et al. 2025 , offering a structured approach to evaluation. Figure 2: Sensitivity of state-of-the-art LMMs to prompt variations. Examples from the MMStar benchmark illustrating divergent model outputs despite identical user queries, caused solely by changes in prompt phrasing ( Left : InternVL-38B, Middle : GPT-4o, Right : Gemini 1.5 Pro). This demonstrates the models’ sensitivity to how instructions are framed. Despite the advantages of MCQ-based evaluations, LMMs exhibit sensitivity to subtle variations in prompt phrasing, raising concerns about the consistency and stability of benchmark results, as reflected in the varied model responses shown in Figure 2 . In this study, we systematically investigate the prompt sensitivity of LMMs by evaluating 8 open-source and 2 proprietary models across 3 multiple-choice question-answering (MCQA) benchmarks covering both image and video modalities. Specifically, we analyze performance variations using 61 systematically designed prompts, categorized into 15 categories and 6 broader supercategories (Figure 1 ). Our goal is to analyse the impact of prompt formulation on model accuracy and benchmark stability, providing insights into best practices for evaluating LMMs on MCQA. The contributions of this paper can be summarized as follows: • Comprehensive Prompt Sensitivity Analysis: We present the most extensive study to date on the impact of prompt variations across diverse multimodal benchmarks and LMM architectures. To facilitate this study, we introduce Promptception, a systematic evaluation framework comprising of 61 prompt types, organized into 15 categories and 6 supercategories, each designed to probe specific aspects of prompt formulation in LMMs. • Evaluation Across Models, Modalities, and Benchmarks: We assess prompt sensitivity across a diverse set of model sizes and architectures, including both open-source and proprietary LMMs. Our analysis spans multiple modalities and benchmarks; MMStar (single image), MMMU-Pro (multi-image), and MVBench (video) and we further evaluate sensitivity across various question dimensions within these benchmarks to ensure a comprehensive understanding. • Best Practices for Prompting: We identify key trends in prompting and propose Prompting Principles for effective and consistent evaluation of LMMs. 2 Experimental Setup 2.1 Visual MCQA Task Definition The MCQA task Robinson et al. 2023 is defined as follows. The LMM is given a question Q , a set of four (or more) choices \mathcal{C}=\{c_{a},c_{b},c_{c},c_{d}\} , exactly one of which is correct (i.e., gold choice c_{g}\in\mathcal{C} ), the prompt P (shown in red) along with the visual (image(s) or video) input V as shown in Figure 3 . Using these inputs, the LMM should give the letter of the correct option a\in\{A,B,C,D\} . Figure 3: Baseline Prompt. This serves as the simplest prompt for the MCQA task. All of our evaluations are zero-shot, with no modifications made to Q , \mathcal{C} , or V . We focus exclusively on zero-shot evaluation, since our primary objective is to assess the impact of prompt variations in isolation. Introducing few-shot examples could reduce the effect of these variations and shift the emphasis toward few-shot learning capabilities, which lies outside the scope of this work. 2.2 Models We evaluate a diverse set of LMMs, including both open-source and proprietary models. The models evaluated are LLaVA-OneVision-7B Li et al. 2025 , Qwen2-VL-7B-Instruct Wang et al. 2024 , InternVL2.5-1B Chen et al. 2025 , InternVL2.5-8B, InternVL2.5-38B, MiniCPM-V-2.6-8B Yao et al. 2024 , Llama-3.2-11B-Vision Patterson et al. 2022 , Molmo-7B-D-0924 Deitke et al. 2024 , GPT-4o ( gpt-4o-2024-08-06 ) OpenAI et al. 2024 , and Gemini 1.5 Pro ( gemini-1.5-pro-latest ) Team et al. 2024 . 2.3 Datasets To evaluate the multimodal reasoning capabilities of LMMs, we use three benchmarks: MMStar Chen et al. 2024 , MMMU-Pro Yue et al. 2024b , and MVBench Li et al. 2024 . These datasets cover single-image, multi-image, and video-based multiple-choice question-answering tasks, assessing different aspects of vision-language understanding. MMStar is a vision-indispensable benchmark designed to eliminate reliance on textual priors and data leakage. It consists of 1,500 carefully curated MCQs that require genuine visual reasoning. MMStar evaluates six core multimodal capabilities across 18 axes and introduces metrics to quantify data leakage and multimodal performance gains. MMMU-Pro is a refined version of the MMMU benchmark Yue et al. 2024a , addressing text-only biases. It contains 1,730 MCQs across