Just Noticeable Difference for Large Multimodal Models

Zijian Chen, Yuan Tian, Yuze Sun, Wei Sun, Zicheng Zhang · 2025

Read Just Noticeable Difference for Large Multimodal Models by Zijian Chen, Yuan Tian, Yuze Sun, Wei Sun, Zicheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00490.txt.

Introduction

Zijian Chen and a team of co-authors published a 2025 study of the smallest visual change that large AI models can notice. For humans, this threshold is called the just noticeable difference. The authors asked whether today’s multimodal models, which combine vision and language, share similar blind spots.

They built a dataset of 21,500 reference images and nearly half a million altered versions across twelve distortion types. Leading models, including GPT-4o, often failed simple comparison questions that humans find easy. The design of both the vision and language parts shaped how sharp the model’s eye was. The authors argue that measuring this threshold matters for safety and efficiency. Their benchmark gives the field a shared yardstick.

Catalog record

CREATORZijian Chen, Yuan Tian, Yuze Sun, Wei Sun, Zicheng Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00490

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Just Noticeable Difference for Large Multimodal Models Thanks: Zijian Chen, Yuan Tian, Zicheng Zhang, and Guangtao Zhai are with the Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai 200240, China, and also with the Shanghai AI Laboratory, Shanghai 200232, China (e-mail: {zijian.chen, ee_tianyuan, zzc1998, zhaiguangtao}@sjtu.edu.cn). ( Corresponding author: Guangtao Zhai. ) Thanks: Yuze Sun and Wenjun Zhang are with the Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai 200240, China (e-mail: {StevenSun, zhangwenjun}@sjtu.edu.cn). Thanks: Wei Sun is with the School of Communication and Electronic Engineering, East China Normal University, Shanghai 200241, China (e-mail: [email protected]). Thanks: Weisi Lin is with the School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798 (e-mail: [email protected]). Zijian Chen, Yuan Tian, Yuze Sun, Wei Sun, Zicheng Zhang Abstract Just noticeable difference (JND), the minimum change that the human visual system (HVS) can perceive, has been studied for decades. Although recent work has extended this line of research into machine vision, there has been a scarcity of studies systematically exploring its perceptual boundaries across multiple tasks and stimulus types, particularly in the current era of rapidly advancing large multimodal models (LMMs), where studying the multifaceted capabilities of models has become a mainstream focus. Moreover, the perceptual defects of LMMs are not investigated thoroughly, resulting in potential security issues and suboptimal response efficiency. In this paper, we take an initial attempt and demonstrate that there exist significant visual blind spots in current LMMs. To systemically quantify this characteristic, we propose a new concept, LMM-JND , together with its determination pipeline. Targeting uncovering the behavior commonalities in HVS-aligned visual perception tasks, we delve into several LMM families and construct a large-scale dataset, named VPA-JND, which contains 21.5k reference images with over 489k stimuli across 12 distortion types, to facilitate LMM-JND studies. VPA-JND exposes areas where state-of-the-art LMMs, including GPT-4o and the InternVL2.5 series, struggle with basic comparison queries and fall significantly short of human-level visual performance. We further explore the effects of vision and language backbones and find a notable correlation between their design philosophy that may instruct the future refinement of LMMs for their visual acuity. Together, our research underscores the significance of LMM-JND as a unique perspective for studying LMMs, and predictable LMM-JND is crucial for security concerns. This work will be available at https://github.com/zijianchen98/LMM-JND . I Introduction Recent years have witnessed unprecedented developments in large multimodal models (LMMs) [ 89 , 91 , 90 , 66 , 4 ] , driven by advancements in architectural innovation, cross-modal alignment techniques, and different training paradigms. Their powerful visual and textual data integration capabilities showcase excellent performance in all-round tasks such as image quality assessment (IQA) [ 80 , 62 ] , visual question answering (VQA) [ 37 , 65 ] , and causal reasoning [ 35 ] . In particular, the recent InternVL3 [ 60 ] series have pushed performance closer to or even exceeded proprietary models. Fig. 1: Illustration of different JND modelings. Mathematically, the fidelity of a signal varies with its content according to a specific function ( e.g. , linear, exponential, or logarithmic). While humans perceive signal quality with benefit zones due to the physiological boundaries. Do LMMs respond likewise? Fig. 2: Examples of the minimal distortion level that can be perceived ( i.e. 1 st JND point). The smallest perceptible levels are shaded in yellow. The red short dash ‘ – ’ indicates that the minimum perceptible range exceeds the maximum distortion level. We can observe that both leading proprietary LMMs (GPT-4o, Gemini 1.5 Pro, and Qwen-VL-Max) and open-source LMM (InternVL2.5-78B) exhibit varying degrees of visual granularity on different stimuli (See a more comprehensive comparison in Tab. V ). Beyond these advancements, few studies have investigated the perception limits of LMMs, which not only include the breadth of their capabilities but also their granularity. As the ultimate receiver and appreciator of the increasingly larger number of visual contents change from the human visual system (HVS) to LMMs, the concerns regarding their perceptual capabilities and safety are gaining prominence. Fig. 2 shows that many LMMs struggle to detect and respond differently to changes that humans can easily perceive. Thus, a question naturally raises: what is the minimal magnitude of changes that LMMs can perceive? Such a characteristic is also known as the just noticeable difference (JND) that has been widely explored with the unique psychological and physiological mechanisms of HVS during the past decades [ 64 , 72 ] . Leveraging the benefit zones in HVS (Fig. 1 ), HVS-JND plays an important role in multimedia data production, representation, and transmission [ 76 , 15 , 58 , 39 ] . More recently, researchers have extended this concept to machine vision system (MVS) [ 6 , 42 , 57 ] and verified the existence of JND on the image classification and object detection tasks, resulting in feasible optimization to machine-oriented visual coding and promoting the implementation of machine vision applications in various scenarios. However, the characteristics obtained from HVS-oriented JND and MVS-oriented JND may not be directly transferred to LMMs due to the following facts. 1) Structural differences of the signal processing pathway: Leaving aside the biological and physiological structures of the HVS, traditional single-modal vision models, such as convolutional neural networks (CNN) and Transformers, differ significantly from current LMMs. The latter augments a foundational visual encoder with a cross-modal alignment module and a decoder-only language generation backbone, which normally have several orders of magnitude of large parameter quantities. 2) Task-specificity: Previous studies [ 42 , 57 ] defined the MVS-JND mainly based on the outcome of simple visual tasks, while the application scope of large models is obviously broader and exhibiting prompt-dependent characteristic. 3) Lack of relevant research: To the best of our knowledge, the existence and characteristics of JND in LMMs remain under-confirmed. Although some works [ 55 , 54 , 56 ] have explored the visual shortcomings of LMMs from the perspectives of perception ability, visual token redundancy, and robustness, their specific magnitude and form have not been systematically investigated. If LMMs have JND, there may exist several potential impacts to take into account. First, finding the LMM-JND points will greatly benefit the processing efficiency of visual content. JND points act as a benefit zone that allows the system to ignore insignificant or irrelevant variations in the input, thus helping compress information, refine the model structure, enhance routine user interactions for online platforms, improve the constantly expanding benchmarks, etc . Meanwhile, such perceptual redundancy features could also explain security concerns, e.g. , data tampering and attacking. Therefore, the presence of LMM-JND precisely indicates the robustness of the LMM in certain scenarios. TABLE I: Comparison of Existing JND-based Datasets with VPA-JND. Combination N Denotes that the Dataset Contains N Different Distortion Types. * Machine Denotes that Only Neural Network Models for Image Classification or Object Detection Tasks are Tested. {\dagger} We Report the Number of Unique Reference Images. All Datasets are Listed in Chronological Order of Publication Dataset Year Stimuli Type # Ref. # Dist. Level # Total Stimuli Resolution Distortion Type Test Object MCL-JCI [ 50 ] 2016 Image 50 100 5,050 1920 \times 1080 JPEG Human MCL-JCV [ 83 ] 2016

Read the complete text