Evaluating Multimodal Large Language Models on Vertically Written Japanese Text
2025
Read Evaluating Multimodal Large Language Models on Vertically Written Japanese Text (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15059.txt.
Introduction
Keito Sasagawa, Shuhei Kurita, and Daisuke Kawahara test how well multimodal language models read Japanese text written vertically. Many Japanese documents run top to bottom, and any serious document tool for Japanese must handle that layout. Yet model testing has mostly used horizontal text, leaving vertical skill unmeasured.
The authors render Japanese passages into images in both directions, building a synthetic reading data set for training and testing. Existing models are then graded on the vertical cases, and fine-tuning effects are checked. The study gives the first focused picture of a gap that matters for real archives, books, and forms in Japan.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-15059
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Multimodal Large Language Models on Vertically Written Japanese Text Abstract Multimodal Large Language Models (MLLMs) have seen rapid advances in recent years and are now being applied to visual document understanding tasks. They are expected to process a wide range of document images across languages, including Japanese. Understanding documents from images requires models to read what are written in them. Since some Japanese documents are written vertically, support for vertical writing is essential. However, research specifically focused on vertically written Japanese text remains limited. In this study, we evaluate the reading capability of existing MLLMs on vertically written Japanese text. First, we generate a synthetic Japanese OCR dataset by rendering Japanese texts into images, and use it for both model fine-tuning and evaluation. This dataset includes Japanese text in both horizontal and vertical writing. We also create an evaluation dataset sourced from the real-world document images containing vertically written Japanese text. Using these datasets, we demonstrate that the existing MLLMs perform worse on vertically written Japanese text than on horizontally written Japanese text. Furthermore, we show that training MLLMs on our synthesized Japanese OCR dataset results in improving the performance of models that previously could not handle vertical writing. The datasets and code are publicly available ( https://github.com/llm-jp/eval_vertical_ja ). Keywords: Multimodal Datasets, Multimodal LLM, Japanese OCR Evaluating Multimodal Large Language Models on Vertically Written Japanese Text Keito Sasagawa ∗ ‡ , Shuhei Kurita † ‡ , Daisuke Kawahara ∗ ‡ ∗ Waseda University, † NII, ‡ NII LLMC Tokyo, Japan [email protected], [email protected], [email protected] Abstract content 1. Introduction Figure 1 : Examples of reading order for Japanese documents (Top: Horizontal writing, Bottom: Vertical writing). The blue numbers attached to each line indicate the order in which the text on that line should be read. In horizontal writing, similar to English documents, characters in each line are read from left to right, and lines are read from top to bottom. In vertical writing, characters in each line are read from top to bottom, and lines are read from right to left. Research on multimodal large language models (MLLMs) has advanced rapidly Bai et al. 2025 ; Zhu et al. 2025 ; Team et al. 2025 ; OpenAI 2025b , and these models are now applied to a wide range of multimodal tasks. Document image question answering is one such task, where a model answers questions about given document images, and various datasets have been proposed Mathew et al. 2021 ; Mathew et al. 2022 ; Tanaka et al. 2021 ; Tanaka et al. 2023 ; Tito et al. 2023 ; Onami et al. 2024 . To solve such tasks, MLLMs are required to read the text within the input document image. A variety of multimodal datasets have been developed for English, and current MLLMs demonstrate high accuracy in reading English text on English-language benchmarks. Japanese documents often include vertically written text, necessitating explicit support for vertical writing. First, we explain the difference in reading order between horizontally and vertically written Japanese text. Figure 1 shows the reading order for horizontally and vertically written Japanese text. In horizontal Japanese writing, as in English, characters are read from left to right and lines progress from top to bottom. In vertical writing, characters are read from top to bottom and lines progress from right to left. It is essential to examine whether models can read vertically written text as well as horizontally written text. However, most Optical Character Recognition (OCR) benchmarks for MLLMs are primarily concerned with horizontally written text, often neglecting the evaluation of vertically written Japanese text. To address this gap, we evaluate the OCR capability of MLLMs for reading vertically written Japanese text. We first build a dataset of synthetic images rendered with Japanese text for training and evaluation. This dataset comprises images containing text in both horizontal and vertical writing and exhibiting multi-column layouts (1-4 columns). We also construct an OCR evaluation dataset from real-world PDF pages that contain vertically written Japanese text. In our experiments, we evaluate multiple open and closed MLLMs, as well as variants fine-tuned on the synthetic image dataset, using our constructed test dataset. We show that existing MLLMs read vertically written Japanese text less accurately than horizontally written text. We further demonstrate that fine-tuning on the synthetic dataset improves models that initially struggle with vertically written Japanese text. The contributions of this study are as follows: 1. We evaluate the Japanese OCR capabilities of MLLMs and quantitatively demonstrate that current MLLMs perform worse with vertical writing than with horizontal writing. 2. We release scripts for synthesizing images of multi-column text in both horizontal and vertical writing styles, along with a dataset of the synthesized images ( JSSODa ). This dataset enables training MLLMs on vertically written text and evaluation. 3. We also release an OCR dataset containing images of vertically written Japanese text from the real-world PDF pages ( VJRODa ). This dataset enables the evaluation of MLLM’s OCR capabilities on real-world document images containing vertically written Japanese text. The datasets and code are publicly available ( https://github.com/llm-jp/eval_vertical_ja ). 2. Related Work 2.1. Multimodal Large Language Models Recent MLLM Liu et al. 2023 ; Liu et al. 2024 ; Li et al. 2025 architectures employ a structure that connects the vision encoder and LLM via a projection layer. It converts the input image into features using a vision encoder, feeds them into a projection layer to transform them into image tokens that the LLM can handle, and then inputs them into the LLM together with text tokens. MLLMs are trained through multimodal instruction tuning, enabling them to handle a wide range of tasks such as document image question answering. Extensive research has been conducted on multimodal models capable of understanding document images Xu et al. 2020 ; Xu et al. 2021 ; Huang et al. 2022 ; Ye et al. 2023a ; Ye et al. 2023b ; Hu et al. 2024 ; Hu et al. 2025 ; Dong et al. 2024 ; Luo et al. 2024 . Among these, notable MLLMs that excel at understanding visual Japanese texts and Japanese document images include Qwen2.5-VL Bai et al. 2025 , InternVL3 Zhu et al. 2025 , and Gemma 3 Team et al. 2025 . Qwen2.5-VL introduces dynamic resolution processing, enabling native handling of images at various resolutions. InternVL3 improves performance by employing techniques such as a pixel unshuffle operation, a dynamic resolution strategy that divides images into multiple tiles, and the variable visual position encoding. Gemma 3 also handles images of various resolutions by dividing the input image into multiple non-overlapping crops of the same size when necessary and feeding them into the vision encoder. 2.2. Japanese OCR Datasets and Vertical Text Datasets NDLOCR NDL Lab 2021 is an OCR program developed by the National Diet Library, Japan (NDL), using materials held in the NDL Digital Collections. For the evaluation of this program, they used in-house evaluation dataset, which is not publicly available. Kindai-OCR Le et al. 2019 is an OCR system for modern Japanese magazines based on an attention-based encoder-decoder architecture. The document images targeted by this system include vertically written Japanese text, but they are somewhat old and not recent documents. SVTD and VTD142