Synthetic Document Question Answering in Hungarian

Jonathan Li, Zoltan Csaki, Nidhi Hiremath, Etash Guha, Fenglu Hong, Edward Ma, Urmish Thakker · 2025

Read Synthetic Document Question Answering in Hungarian by Jonathan Li, Zoltan Csaki, Nidhi Hiremath, Etash Guha, Fenglu Hong, Edward Ma, Urmish Thakker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23008.txt.

Introduction

Jonathan Li and co-authors build Hungarian document question answering data sets in two thousand twenty-five. Vision-language models ace English document questions but stumble in smaller languages for lack of training and test data. Hungarian, a mid-sized web language, is their test case.

One set was curated by hand from Hungarian pages in Common Crawl; a larger one was machine-made from the same source and filtered hard to near human quality. A third set holds one hundred seventeen thousand Hungarian PDF pages with transcripts, fit for training text readers. Fine-tuning a mid-size open model on a blend lifted its score by over seven points. All sets and code are public, to seed work in more languages.

Catalog record

CREATORJonathan Li, Zoltan Csaki, Nidhi Hiremath, Etash Guha, Fenglu Hong, Edward Ma, Urmish Thakker
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23008

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Synthetic Document Question Answering in Hungarian Jonathan Li, Zoltan Csaki, Nidhi Hiremath, Etash Guha, Fenglu Hong, Edward Ma, Urmish Thakker Abstract Modern VLMs have achieved near-saturation accuracy in English document visual question-answering (VQA). However, this task remains challenging in lower resource languages due to a dearth of suitable training and evaluation data. In this paper we present scalable methods for curating such datasets by focusing on Hungarian, approximately the 17th highest resource language on the internet Nguyen et al. 2023 . Specifically, we present HuDocVQA and HuDocVQA-manual, document VQA datasets that modern VLMs significantly underperform on compared to English DocVQA Mathew et al. 2021 . HuDocVQA-manual is a small manually curated dataset based on Hungarian documents from Common Crawl Crawl 2025 , while HuDocVQA is a larger synthetically generated VQA data set from the same source. We apply multiple rounds of quality filtering and deduplication to HuDocVQA in order to match human-level quality in this dataset. We also present HuCCPDF, a dataset of 117k pages from Hungarian Common Crawl PDFs along with their transcriptions, which can be used for training a model for Hungarian OCR. To validate the quality of our datasets, we show how finetuning on a mixture of these datasets can improve accuracy on HuDocVQA for Llama 3.2 11B Instruct by +7.2%. We release our datasets and code to foster further research in multilingual DocVQA. \caption@setdefaultlabelsep period 1 Introduction Figure 1 : An example question-answer pair from HuDocVQA-manual As large language models (LLMs) quickly reach saturation and human-level performance on a host of traditional NLP tasks, vision-language models (VLMs) are quickly reaching saturation on traditional multimodal benchmarks as well. One such benchmark is DocVQA Mathew et al. 2021 , where human-level performance is estimated at 94.36% exact-match accuracy and 0.981 ANLS Goyal et al. 2017 . Recent open-weights VLMs such as Llama 3.2 90B Instruct Grattafiori et al. 2024 and Qwen 2.5 72B Instruct Bai et al. 2025 , as well as closed-weights VLMs in GPT-4o and Claude 3.7 Sonnet have achieved well over 0.9 ANLS. However, while LLM training and evaluation has diversified and shown proficiency in a host of non-English and even low-resource languages Grattafiori et al. 2024 ; Kolavi 2024 ; Wang et al. 2024 ; Qwen et al. 2025 ; Yue et al. 2024 , there are comparatively fewer studies on the multilingual capabilities of VLMs. In particular, multilingual DocVQA suffers from a surprising lack of training and test data, with a major pain point being the scalability of dataset creation efforts. Existing multilingual VQA works fail to address these issues: EXAMS-V Das et al. 2024 and JDocQA Onami et al. 2024 involved painstaking manual curation and quality checking by native speakers, while MaxM Changpinyo et al. 2022 applies machine translation to captioning data, which may suffer from “translationese” Riley et al. 2020 . A more detailed discussion on related work can be found in Appendix 5 . In this paper, we take advantage of the multilingual capabilities of LLMs to scalably construct multilingual DocVQA training and evaluation data. In Section 2 , we describe the dataset curation process, including sourcing, synthetic data generation, and quality filtering for 3 Hungarian multimodal datasets: HuDocVQA-manual, HuDocVQA, and HuCCPDF. In Section 3 , we show how modern VLMs significantly underperform on HuDocVQA and HuDocVQA-manual compared to English DocVQA, and how finetuning on HuDocVQA and HuCCPDF can improve accuracy by up to +7.2% for existing strong VLMs like Llama 3.2 11B Instruct. We plan to publicly release our datasets and code to further foster scalable construction of multilingual document datasets. 2 Dataset Curation 2.1 HuDocVQA-manual To create HuDocVQA-manual, the authors annotated 54 pages of Hungarian PDFs from Common Crawl to serve as a human-verified benchmark for document VQA in Hungarian. For each page, we used Google Translate and DeepL to read the document and create a semantically meaningful question-answer pair in English. Then, we translated the English pair back into Hungarian with the same tools. All examples were then verified by a native Hungarian speaker. 2.2 HuDocVQA Figure 2 : A diagram of our synthetic data pipeline In this section, we describe each step of our synthetic question-answer generation pipeline for HuDocVQA. See Fig. 2 for a diagram of the full process. 2.2.1 Synthetic VQA Generation To generate synthetic VQA pairs, we start with a large collection of PDFs in the target language, as described in Section 2.3 , totaling 24,341 pages of Hungarian text. We treat each page as a single image and apply two methods of text extraction: OCR using pytesseract ( Hoffstaetter and contributors 2025 ) and PDF parsing with PyMuPDF ( McKie. and contributors 2025 ) . This helps mitigate biases introduced by each text extraction method. Once text is extracted from a page, we construct a one-shot prompt using one of three randomly sampled human-written Hungarian QA pairs. We then use LLaMA 3.3 70B ( Grattafiori et al. 2024 ) to generate new QA pairs based on the one-shot prompt and the extracted text. For each image, we generate 4 QA pairs (2 from the PyTesseract text and 2 from the PyMuPDF text) and discard any model outputs that do not follow the question and answer format of the few-shot examples. For generation and prompting details, see Section 9 . In total, we generate 93,933 QA pairs, averaging 3.9 questions per image. In addition, we create training, test, and validation splits based on each image. See Table 1 for full dataset statistics. Dataset Split # Images # QAs HuDocVQA-manual Test 54 54 HuDocVQA Train 21,800 84,192 Validation 1,283 4,930 Test 1,258 4,811 + filtering Train 20,059 55,325 Validation 1,183 3,382 Test 1,162 3,315 Table 1 : HuDocVQA Statistics 2.2.2 Filtering VQA Data In our initial observations of the generated synthetic VQA data, we identified several recurring issues. For each issue, we developed quality checks to filter out examples containing these issues. Table 2 summarizes these common issues along with the heuristic filters applied to mitigate them, and Table 5 shows the number of examples removed out by each heuristic filter. In total, we filter out 31,911 QA pairs, leaving 62,022 remaining to comprise HuDocVQA. Issue Filter The text in the image is too short (e.g., only a title). Text length filter: text must exceed 60 characters The question is not related to the text in the image. N-gram overlap between question and text (n=4), with a 12% overlap threshold The question is generated in the wrong language. Language filter using Python’s langdetect ( Danilak 2018 ) Duplicate questions are generated for the same image. Use LLM to detect whether any two questions for the same image are paraphrases Table 2 : Issues identified in our synthetic data and the corresponding filters we applied. 2.3 HuCCPDF In order to address the shortage of high-quality datasets in Hungarian for both OCR and visual question answering, we filtered PDFs from Common Crawl Crawl 2025 to create HuDocVQA-manual, HuDocVQA, and HuCCPDF. The same procedure was applied to collect PDFs for all 3 datasets, with image-level deduplication applied between all 3. Specifically for HuCCPDF, we processed 3750/60,000 WARC Consortium 2025 files from CC-MAIN-2021-17 . We collect links to PDFs from each WARC file, download them, and extract their

Read the complete text