Revisiting MLLM Based Image Quality Assessment: Errors and Remedy

Zhenchen Tang, Songlin Yang, Bo Peng, Zichuan Wang, Jing Dong · 2025

Read Revisiting MLLM Based Image Quality Assessment: Errors and Remedy by Zhenchen Tang, Songlin Yang, Bo Peng, Zichuan Wang, Jing Dong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07812.txt.

Introduction

Zhenchen Tang, Songlin Yang, and Bo Peng fix a mismatch when multimodal models grade image quality. Quality scores are continuous, but language models answer in discrete tokens. Converting word levels like good into numbers introduces errors, and level words blur meaning, dragging down related abilities.

An error analysis names the failure modes, then a remedy reworks how scores are produced. Keeping the model strengths while extracting smooth scores is the goal. Automated quality assessment feeds photo tools, compression, and medical imaging triage. A judge that confuses its own scale helps none of those, so aligning output form with task form matters.

Catalog record

CREATORZhenchen Tang, Songlin Yang, Bo Peng, Zichuan Wang, Jing Dong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07812

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting MLLM Based Image Quality Assessment: Errors and Remedy Zhenchen Tang, Songlin Yang, Bo Peng, Zichuan Wang, Jing Dong Abstract The rapid progress of multi-modal large language models (MLLMs) has boosted the task of image quality assessment (IQA). However, a key challenge arises from the inherent mismatch between the discrete token outputs of MLLMs and the continuous nature of quality scores required by IQA tasks. This discrepancy significantly hinders the performance of MLLM-based IQA methods. Previous approaches that convert discrete token predictions into continuous scores often suffer from conversion errors. Moreover, the semantic confusion introduced by level tokens (e.g., “good”) further constrains the performance of MLLMs on IQA tasks and degrades their original capabilities to related tasks. To tackle these problems, we provide a theoretical analysis of the errors inherent in previous approaches and, motivated by this analysis, propose a simple yet effective framework, Q-Scorer. This framework incorporates a lightweight regression module and IQA-specific score tokens into the MLLM pipeline. Extensive experiments demonstrate that Q-Scorer achieves state-of-the-art performance across multiple IQA benchmarks, generalizes well to mixed datasets, and further improves combined with other methods. Code — https://github.com/2kxx/Q-Scorer 1 Introduction Image Quality Assessment (IQA) is a fundamental task in computer vision, aiming to quantify the perceptual quality of images in a way that aligns closely with human visual perception. With the recent advances in reinforcement learning-based post-training paradigms, IQA has become increasingly important as a reward signal for downstream tasks, such as image generation ( 20 ; 44 ; 19 ; 18 ; 8 ) and enhancement ( 48 ; 49 ; 35 ) . Figure 1: Overview of MLLM-based IQA methods and error analysis. The figure shows how MLLMs are adapted for label conversion and score prediction, and highlights steps causing conversion errors and semantic confusion. Recently, leveraging MLLMs ( 22 ; 40 ; 1 ; 5 ) to improve IQA performance has shown promise, as MLLMs can understand visual images using linguistic intelligence, which greatly facilitates aligning model predictions with human evaluations across diverse image types. However, existing MLLM-based IQA methods ( 36 ; 38 ; 37 ) typically formulate the task as learning the correlation between images and discrete textual tokens (e.g., “poor,” “good”), which fits the token output nature of MLLMs but conflicts with the intrinsic nature of IQA — predicting continuous quality scores. As shown in Fig. 1 , although several studies ( 38 ; 41 ; 17 ) has explored techniques to bridge this mismatch, two key problems remain: (1) Conversion Errors: Existing token-based approaches suffer from inevitable theoretical errors during the label conversion process. Since real-valued Mean Opinion Scores (MOSs) are quantized into discrete tokens, even perfect classification cannot exactly recover the ground-truth MOS. We theoretically analyze this issue and prove that such errors are intrinsic to token-based IQA formulations in Sec. 3.1 . (2) Semantic Confusion: Most quality-related textual descriptions (i.e., level tokens) used in current methods are drawn from pre-trained LLM vocabularies, whose semantics are not optimized for IQA. This introduces two problems: pre-tuning semantic confusion, where these tokens carry prior meanings unrelated to visual quality; post-tuning semantic confusion, where training on IQA-specific tasks may overwrite the semantic integrity of these tokens. To address these problems, we propose Q-Scorer , a simple yet effective framework that incorporates a lightweight regression module and IQA-specific score tokens into the MLLM pipeline. For conversion errors, we adopt an MLP-based module to regress the continuous IQA scores. Although prior work used a linear layer on the final token ( 9 ) , its entanglement with IQA-irrelevant information limits performance compared to token-based methods (even the baseline Q-Align), which causes insufficient embedding expressiveness. To address this issue and mitigate semantic confusion, we introduce a set of IQA-specific score tokens. These tokens guide the MLP module to improve IQA score prediction without compromising the original capabilities of MLLMs. Equipped with these designs, our method achieves state-of-the-art performance using only LoRA-based fine-tuning on limited number of model parameters. Q-Scorer consistently outperforms baseline approaches in score regression, and despite being trained solely on the KonIQ dataset, it also exhibits strong generalization to other datasets. Furthermore, it generalizes well to mixed-dataset settings and can be seamlessly integrated with other methods to achieve even better performance. Our contributions can be summarized as follows: • We conduct error analysis and find that existing MLLM-based IQA methods suffer from conversion errors and semantic confusion when using discrete level tokens. • To address these challenges, we propose Q-Scorer, a simple yet effective framework that leverages IQA-specific score tokens and an MLP regressor. • Extensive experiments demonstrate that our framework achieves state-of-the-art performance across multiple benchmarks, generalizes well to mixed datasets, and further improves combined with other methods. 2 Related Works 2.1 Traditional IQA Methods Prior to the emergence of MLLMs, traditional IQA methods were generally categorized into full-reference (FR) and no-reference (NR) approaches. Classical FR methods rely on handcrafted similarity metrics that compare a distorted image with its high-quality reference, such as PSNR ( 10 ) and SSIM ( 34 ) . In contrast, NR methods estimate perceptual quality directly from a single image using natural scene statistics, as exemplified by BRISQUE ( 25 ) and NIQE ( 26 ) . With the advancement of deep learning, data-driven IQA approaches have achieved significant improvements by directly regressing human perceptual scores from raw image inputs ( 2 ; 30 ; 29 ) . More recent studies further boost performance and generalization by incorporating multi-scale features ( 14 ) , co-training on multiple datasets ( 46 ) , multitask learning ( 47 ) , or leveraging pretrained vision-language models such as CLIP ( 33 ; 31 ) . 2.2 MLLM-Based IQA methods MLLM-based IQA methods leverage the foundational knowledge embedded in MLLMs to improve IQA performance and enhance generalization. Q-Bench ( 36 ) demonstrates that MLLMs possess low-level visual perception and understanding, enabling them to predict quantifiable quality scores. Q-Instruct ( 37 ) improves question-answering accuracy including IQA tasks, by introducing instruction-response style description dataset. DepictQA ( 43 ) and its extension ( 42 ) generates detailed quality assessments by training on large-scale explanation dataset. Moreover, Q-Ground ( 3 ) and SEAGULL ( 4 ) extend IQA to more fine-grained tasks, including visual quality grounding and region-aware scoring. 2.3 IQA Scorers Although numerous MLLM-based IQA methods have emerged recently, their scoring strategies are limited and can only be categorized into token-based and regression-based approaches , depending on whether they utilize an MLP ( 17 ) . We refer to the methods proposing these scoring strategies as IQA Scorers. First are token-based methods. Initially, Q-Bench ( 36 ) introduces a binary softmax strategy to predict quality scores by two discrete levels. Compare2Score ( 50 ) adopts a pairwise comparison strategy to infer image quality scores. Building on Q-Bench, the widely adopted method Q-Align ( 38 ) improves scoring process by discretizing quality into five levels and training MLLMs using one-hot labels. DeQA-Score ( 41 ) further enhances Q-Align by introducing soft labels based on score distributions. RealQA ( 17 ) takes a different approach by directly predicting two extra significant digits of the score. In contrast, regression-based methods are rarely explored in MLLM-based IQA. While MLPs are commonly

Read the complete text