Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan Conference: ; August; 2025 Preprint CCS: Computing methodologies Model verification and validation CCS: Computing methodologies Model verification and validation

Jui-Ming Yao, Bing-Cheng Xie, Sheng-Wei Peng, Hao-Yuan Chen, He-Rong Zheng, Bing-Jia Tan, Peter Shaojui Wang, Shun-Feng Su · 2025

Read Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan Conference: ; August; 2025 Preprint CCS: Computing methodologies Model verification and validation CCS: Computing methodologies Model verification and validation by Jui-Ming Yao, Bing-Cheng Xie, Sheng-Wei Peng, Hao-Yuan Chen, He-Rong Zheng, Bing-Jia Tan, Peter Shaojui Wang, Shun-Feng Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01274.txt.

Introduction

Jui-Ming Yao and co-authors launch a test set for multimodal models in Traditional Chinese in a 2025 paper. Most test sets skip triple input checks in this script and ignore response speed. Their set, Multi-TW, pairs image or sound with text in 900 multiple choice questions drawn from official Huayu skill tests in Taiwan.

The team ran both any to any models and vision language models fed with sound transcripts. Closed source models beat open ones on most tasks, though open models held their own on sound. Direct any to any runs also answered faster than split pipelines. The authors call for more tuning in Traditional Chinese.

Catalog record

CREATORJui-Ming Yao, Bing-Cheng Xie, Sheng-Wei Peng, Hao-Yuan Chen, He-Rong Zheng, Bing-Jia Tan, Peter Shaojui Wang, Shun-Feng Su
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01274

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan Conference: ; August; 2025 Preprint CCS: Computing methodologies Model verification and validation CCS: Computing methodologies Model verification and validation Jui-Ming Yao, Bing-Cheng Xie, Sheng-Wei Peng, Hao-Yuan Chen, He-Rong Zheng, Bing-Jia Tan, Peter Shaojui Wang, Shun-Feng Su Abstract . Multimodal Large Language Models (MLLMs) process visual, acoustic, and textual inputs, overcoming the limitations of single-modality LLMs. However, existing benchmarks often neglect tri-modal evaluation in Traditional Chinese and overlook inference latency. To fill this gap, we introduce Multi-TW , the first Traditional Chinese benchmark for evaluating the performance and latency of any-to-any multimodal models. Multi-TW comprises 900 multiple-choice questions (image & text, audio & text pairs) from authentic proficiency tests developed with the Steering Committee for the Test of Proficiency-Huayu (SC-TOP). We evaluated various any-to-any models and vision-language models (VLMs) with audio transcription. Our findings show closed-source models generally outperform open-source ones across modalities, though open-source models can excel in audio tasks. End-to-end any-to-any pipelines demonstrate significant latency advantages over VLM with separate audio transcription. Multi-TW offers a holistic view of model capabilities, underscoring the need for Traditional Chinese fine-tuning and efficient multimodal architectures. 1. Introduction Pre-trained Large Language Models (LLMs), such as LLaMA ( Touvron et al. 2023a ; Touvron et al. 2023b ) and Qwen ( Bai et al. 2023 ; Yang et al. 2024 ; Qwen et al. 2025 ) , have demonstrated remarkable success across a wide range of natural language processing (NLP) tasks. However, these text-only models remain constrained by their single-modality input. To address this limitation, recent research has increasingly focused on Multimodal Large Language Models (MLLMs), which can jointly process and reason over visual, acoustic, and textual inputs ( Liang et al. 2024 ; Yin et al. 2024 ) . In the visual domain, models such as CLIP ( Radford et al. 2021 ) and Flamingo ( Alayrac et al. 2022 ) have shown that contrastive pretraining and multimodal fusion architectures enable state-of-the-art zero-shot image classification, image captioning, and few-shot visual reasoning ( Li et al. 2025b ; Islam et al. 2023 ) . Building upon these breakthroughs, Vision-Language Models (VLMs) like LLaVA ( Liu et al. 2024 ) have pushed the frontier further, inspiring fine-tuned successors such as Vicuna ( Zheng et al. 2023 ) and Alpaca ( Taori et al. 2023 ) , which expand multimodal reasoning capabilities across broader task domains. The models evaluated in our experiments, such as the LLaVA series, PaliGemma 2 ( Steiner et al. 2024 ) , Idefics2 ( Laurençon et al. 2024 ) , Llama 3.2-Vision ( Meta 2024 ) , UI-TARS ( Qin et al. 2025 ) and Qwen VL ( Bai et al. 2025 ) series, represent the cutting edge in these developments. With the evolution of VLMs, increasing attention has turned toward audio-language modeling. Audio Language Models (ALMs) typically employ an audio encoder that transforms raw waveform signals into token representations that can be processed by a language model ( Latif et al. 2023 ; Peng et al. 2025 ) . For instance, Qwen-Audio ( Chu et al. 2023 ) and Qwen-Audio2 ( Chu et al. 2024 ) utilize the Qwen model series ( Bai et al. 2023 ; Yang et al. 2024 ) as their language modeling backbone and incorporate OpenAI’s Whisper ( Radford et al. 2022 ) for end-to-end speech recognition. Other architectures, such as AudioPaLM ( Rubenstein et al. 2023 ) , fuse the text-based capabilities of PaLM-2 ( Anil et al. 2023 ) with the discrete audio token modeling of AudioLM ( Borsos et al. 2023 ) , enabling both high-quality speech recognition and speech-to-speech translation in a unified framework. More recently, research has progressed toward universal any-to-any multimodal models that support cross-modal input and output across vision, audio, and text. Prominent examples include NExT-GPT ( Wu et al. 2024 ) , AnyGPT ( Zhan et al. 2024 ) and Unified-IO 2 ( Lu et al. 2023 ) , all pushing the limits of unified multimodal intelligence. Later, this trend transfer into multilingual support, as shown in open-source models like Baichuan-Omni-1.5 ( Li et al. 2025a ) and Qwen2.5-Omni ( Xu et al. 2025 ) , as well as closed-source systems such as gemini, which achieve strong performance in both Chinese and English understanding. To rigorously evaluate the capabilities of such models, several benchmarks have been proposed. However, most evaluations still assess only two modalities at a time. For instance, NExT-GPT ( Wu et al. 2024 ) and AnyGPT ( Zhan et al. 2024 ) focus on pairwise modality evaluations. Recently, Qwen2.5-Omni ( Xu et al. 2025 ) and Baichuan-Omni-1.5 ( Li et al. 2025a ) have adopted OmniBench ( Li et al. 2025c ) , a tri-modal benchmark designed to assess performance across text, image, and audio simultaneously, providing deeper insight into a model’s unified reasoning ability. Despite these advances, a critical gap remains in the evaluation of multimodal models in Traditional Chinese. Existing Traditional Chinese benchmarks are largely text-based. TMMLU ( Hsu et al. 2023 ) and its extension TMMLU+ ( Tam et al. 2024 ) provide comprehensive text-only evaluations of LLMs. VisTW ( Tam et al. 2025 ) moves into the multimodal space by evaluating VLMs on multiple-choice and dialog-based tasks; however, no benchmark currently supports comprehensive evaluation across textual, visual, and acoustic modalities in Traditional Chinese. In addition to this linguistic gap, we observe that most existing benchmarks prioritize accuracy, often overlooking model inference time . This approach is insufficient for real-world applications where both accuracy and efficiency are crucial. To address this gap, we introduce Multi-TW , the first benchmark specifically designed for evaluating the performance and latency of any-to-any multimodal models in Traditional Chinese. Multi-TW consists of image-text and audio-text pairs, enabling rigorous evaluations that cover textual, visual, and acoustic modalities. All datasets are open-sourced and available at: https://drive.google.com/drive/folders/1IvBOXR1GpMNtst0T3HT6dM59_ASlXdyn . In summary, our contributions are as follows: • We propose Multi-TW , the first Traditional Chinese benchmark for rigorous evaluation across text, audio, and visual inputs. • We collaborated with the Steering Committee for the Test of Proficiency-Huayu to incorporate authentic, real-world assessment tasks into our machine evaluation framework. • We conduct comprehensive experiments on both any-to-any models and VLMs (the latter using ASR for audio input). • In addition to accuracy, we evaluate latency to offer a more holistic view of model performance in real-world settings. 2. Multi-TW Benchmark In this section, we provide a concise overview of Multi-TW, including its construction process, validation procedures, and data format specifications to support reproducibility. Our dataset is derived from real-world exams, detailed further in Section 2.2 . 2.1. Data Construction To construct the Multi-TW dataset, we collaborated with the Steering Committee for the Test of Proficiency-Huayu (SC-TOP), a dedicated agency responsible for developing and promoting Taiwan’s Mandarin proficiency tests for non-native speakers. These exams, primarily in a multiple-choice format, underwent rigorous utility analysis to ensure their practical value and effectiveness. Figure 1. Illustration of data collection interface. Construction . The construction phase spanned from September 2023 to December 2023, primarily using publicly available data. All items in Multi-TW underwent a standardized collection and processing workflow performed by our research team to ensure consistency and

Read the complete text