Judge Model for Large-scale Multimodality Benchmarks

Min-Han Shih, Yu-Hsin Wu, Yu-Wei Chen · 2026

Read Judge Model for Large-scale Multimodality Benchmarks by Min-Han Shih, Yu-Hsin Wu, Yu-Wei Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06106.txt.

Introduction

Grading multimodal AI systems at scale needs a judge that can explain itself. Min-Han Shih and 2 co-authors propose such a Judge Model in a 2026 paper. Their benchmark spans text, audio, image, and video tasks, drawn from public datasets with fixed sampling to keep tests reproducible and reduce leakage.

Instead of a bare score, the framework combines judgments across modalities, checks the quality and consistency of a model's reasoning, and produces diagnostic feedback. Several leading multimodal models were evaluated on 280 samples. The judge's assessments aligned strongly with human annotators. The authors present the system as a scalable, interpretable evaluation pipeline for future multimodal research.

Catalog record

CREATORMin-Han Shih, Yu-Hsin Wu, Yu-Wei Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06106

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Judge Model for Large-scale Multimodality Benchmarks Min-Han Shih, Yu-Hsin Wu, Yu-Wei Chen Abstract We propose a dedicated multimodal Judge Model designed to provide reliable, explainable evaluation across a diverse suite of tasks. Our benchmark spans text, audio, image, and video modalities, drawing from carefully sampled public datasets with fixed seeds to ensure reproducibility and minimize train–test leakage. Instead of simple scoring, our framework aggregates multimodal judgments, analyzes the quality and reasoning consistency of model outputs, and generates diagnostic feedback. We evaluate state-of-the-art MLLMs, including Gemini-2.5-flash, Phi-4, and Qwen-2.5-omni, across 280 multimodal samples and compare judge-model assessments with human annotators. Results show strong alignment between the Judge Model and human scores, demonstrating its potential as a scalable, interpretable evaluation pipeline for future multimodal AI research. 1 Introduction Recent advances in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have demonstrated remarkable progress in understanding and reasoning across multiple modalities such as text, image, audio, and video. Models like GPT-5 , Gemini 2.5, and Claude 4 sonnet can now interpret complex multimodal inputs, perform reasoning over long contexts, and generate structured, contextually aware outputs. However, as the capability and diversity of MLLMs expand, objectively and efficiently evaluating their performance has become an increasingly challenging problem. Traditional evaluation pipelines rely on either human annotation or static benchmark scoring, which are often time-consuming, inconsistent, and limited to task-specific performance. Meanwhile, recent approaches, such as LLM-as-a-Judge, leverage large models as evaluators. 6 ; 14 ; 16 Although these methods reduce human workload, they introduce several issues: (1) lack of multimodal understanding, since many judge systems are purely text-based; and (2) insufficient explainability and reliability when compared to human judgment on complex reasoning or cross-modal alignment tasks. To address these limitations, our project proposes a Judge Model designed specifically for large-scale multimodal benchmarks. Instead of scoring, our framework aggregates and calibrates judgments across multiple perspectives. The Judge will evaluate the outputs of various MLLMs, such as Gemini 2.5, Phi-4 1 , and Qwen-2.5 38 , analyzing the quality of the response, the alignment of the modality and the consistency of the reasoning trace. Beyond scoring, it will also generate comprehensive feedback and guidance, aiming to establish an interpretable and reliable evaluation pipeline for multimodal AI systems. 2 Related Work 2.1 Reasoning LLMs LLMs know when to think step-by-step and only generate step-by-step reasoning when necessary. 47 Prior works on Chain-of-Thought (CoT) mostly focus on the correctness of the CoT reasoning steps or whether the reasoning steps are faithful to the question and support the final answer. 51 ; 18 2.2 LLM-as-the-Judge LLM-based evaluators are LLMs that are prompted to judge the quality of some samples based on specific criteria. LLM-based evaluators evaluate the quality of a single sample using a score such as Likert scores 30 on a scale of 1 to 5. 8 ; 9 ; 10 ; 11 ; 53 LLM-based evaluators can also compare the quality of a pair of samples and judge which one is better. 53 2.3 Multimodal Benchmark Vision benchmark expands from image benchmarks to video benchmarks that introduce temporal structure and richer scene context 37 , such as Vision Quesion-Answering 3 ; 19 , chart understanding 33 ; 46 ; 34 , and general capability suites 17 ; 29 ; 31 . Different video datasets emphasize different abilities: temporal ordering 5 ; 32 , procedural understanding 42 ; 48 , egocentric perception 20 ; 13 , and world modeling 21 . Audio benchmarks such as AudioSet 27 and VGGSound 7 target event-level acoustic classification, while speech datasets cover specific facets of communication: VoxCeleb 36 ; 12 for speaker identity, AudioCaps / SpeechCaps 27 ; 24 for audio/speech captioning, LRS 2 for transcription, and AVA-ActiveSpeaker 28 for frame-level speaking and audibility labels. Dynamic-SUPERB 23 ; 22 proposed a dynamic, collaborative benchmark in speech and audio tasks. 3 Datasets 3.1 Text-to-Text Datasets To build a compact, reproducible benchmark, we pull small, fixed-size samples from widely used public datasets on HuggingFace Hub. For each dataset we explicitly name the split used so that readers can assess the risk of train–test leakage in future model training. We sample without replacement using a fixed random seed. The data source and the number of each task are listed in Table 1 . Task Family Dataset (HF ID) Split #Items Answer Type Reasoning–Code nvidia/OpenCodeReasoning split_0 20 code / final string Reasoning–Math nvidia/OpenMathReasoning cot 20 generative (final value) Expert MCQ Idavidrein/gpqa ( fallback: fingertap/GPQA-Diamond ) test 10 multiple choice Reading Comprehension ucinlp/drop validation 20 extractive span Commonsense Reasoning Rowan/hellaswag validation 15 multiple choice Commonsense Reasoning jet-ai/social_i_qa validation 15 multiple choice Instruction Following google/IFEval train 15 generative (constrainted) Instruction Following YuxinJiang/FollowBench train ( fallback: validation ) 15 generative (constrainted) Total 130 Table 1: Datasets, splits, counts, and answer types used in our text benchmark. 3.2 Text-to-Text Tasks 1 . Reasoning–Code: program synthesis problem solving requiring structured outputs or final values. 4 2. Reasoning–Math: multi-step mathematical reasoning aiming at a final canonical answer. 35 3. Expert MCQ: graduate-level STEM multiple-choice questions 39 ; 50 ; 25 . 4. Reading Comprehension: passage-based QA with extractive answers 15 5. Commonsense Reasoning: everyday physical/social reasoning in multiple-choice format. 52 ; 40 6. Instruction Following: adherence to explicit constraints and formatting instructions. 26 ; 54 3.3 Audio-Language Datasets Our audio-language datasets are constructed from four open-source datasets hosted on Hugging Face. These datasets were carefully selected to ensure diversity across audio modalities and task types, including environmental sound understanding, bird sound classification, music generation, and speech question answering. The sources and task types are summarized in Table 2 . Task Family Dataset (HF ID) Split #Items Audio Captioning kuanhuggingface/audiocaps_hallucination test 10 Birdsound Detection tglcourse/5s_birdcall_samples_top20 validation 10 Music Style Detection nyuuzyou/suno train 10 Speech Question-Answering AudioLLMs/public_sg_speech_qa_test test 20 Total 50 Table 2: Datasets, splits, counts, and answer types used in our audio benchmark. 3.4 Audio-Language Tasks 1. Question–Answer: Involve recognizing and describing the presence or absence of specific sounds, or answering questions based on audio content. 41 ; 27 ; 43 2. Instruction–Following: Evaluate the model’s ability to interpret and execute textual instructions for auditory inputs. 43 3.5 Video Datasets Our video dataset comes from the Hugging Face public dataset. It consists of AI-generated short clips created by generative video models such as Sora, Veo2, and Kling. Each clip is paired with human-curated questions designed to test the reasoning of a model about common sense and physical consistency. To ensure reproducibility, we sample using a fixed random seed and select a total of 50 video clips from non-training sets, which minimizes the chance of overlap with materials that LLMs might have already seen during pretraining. This subset provides a controlled and diverse evaluation set for testing video content understanding and hallucination robustness in MLLMs. The sources and task types are summarized in Table 3 . Dataset (HF ID) Split #Items IntelligenceLab/VideoHallu validation 25 IntelligenceLab/VideoHallu test 25 Total 50 Table 3: Datasets, splits, counts, and answer types used in our video benchmark. 3.6 Video Tasks Our task focuses on common sense and physical reasoning rather than traditional visual recognition, due to the use of AI-generated synthetic videos. The model is required to analyze each short clip, interpret the given question, and determine whether

Read the complete text