Understanding the World’s Museums through Vision-Language Reasoning
Ada-Astrid Balauca, Sanjana Garai, Stefan Balauca, Rasesh Udayakumar Shetty, Naitik Agrawal, Dhwanil Subhashbhai Shah, Yuqian Fu, Xi Wang, Kristina Toutanova, Danda Pani Paudel, Luc Van Gool · 2024
Read Understanding the World’s Museums through Vision-Language Reasoning by Ada-Astrid Balauca, Sanjana Garai, Stefan Balauca, Rasesh Udayakumar Shetty, Naitik Agrawal, Dhwanil Subhashbhai Shah, Yuqian Fu, Xi Wang, Kristina Toutanova, Danda Pani Paudel, Luc Van Gool (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01370.txt.
Introduction
Ada-Astrid Balauca leads a team of eleven authors in 2024 in teaching machines to understand museum exhibits from pictures and questions. Museums hold carefully documented objects from many eras and cultures, and visitors naturally ask questions that call for expert knowledge.
Answering well means more than naming what a photo shows; the system must link visual details to history and context. The authors build on vision-language reasoning, where a model reads both image and text together. Large, well-structured museum collections can train such specialist models. The work points toward guides that answer visitor curiosity with real depth.
Catalog record
| CREATOR | Ada-Astrid Balauca, Sanjana Garai, Stefan Balauca, Rasesh Udayakumar Shetty, Naitik Agrawal, Dhwanil Subhashbhai Shah, Yuqian Fu, Xi Wang, Kristina Toutanova, Danda Pani Paudel, Luc Van Gool |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01370
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding the World’s Museums through Vision-Language Reasoning Ada-Astrid Balauca, Sanjana Garai, Stefan Balauca, Rasesh Udayakumar Shetty, Naitik Agrawal, Dhwanil Subhashbhai Shah, Yuqian Fu, Xi Wang, Kristina Toutanova, Danda Pani Paudel, Luc Van Gool Abstract Museums serve as vital repositories of cultural heritage and historical artifacts spanning diverse epochs, civilizations, and regions, preserving well-documented collections. Data reveal key attributes such as age, origin, material, and cultural significance. Understanding museum exhibits from their images requires reasoning beyond visual features. In this work, we facilitate such reasoning by (a) collecting and curating a large-scale dataset of 65M images and 200M question-answer pairs in the standard museum catalog format for exhibits from all around the world; (b) training large vision-language models on the collected dataset; (c) benchmarking their ability on five visual question answering tasks. The complete dataset is labeled by museum experts, ensuring the quality as well as the practical significance of the labels. We train two VLMs from different categories: the BLIP Li et al. 2022c model, with vision-language aligned embeddings, but lacking the expressive power of large language models, and the LLaVA Liu et al. 2023 model, a powerful instruction-tuned LLM enriched with vision-language reasoning capabilities. Through exhaustive experiments, we provide several insights on the complex and fine-grained understanding of museum exhibits. In particular, we show that some questions whose answers can often be derived directly from visual features are well answered by both types of models. On the other hand, questions that require the grounding of the visual features in repositories of human knowledge are better answered by the large vision-language models, thus demonstrating their superior capacity to perform the desired reasoning. Find our dataset, benchmarks, and source code at: github.com/insait-institute/Museum-65 1 Introduction Figure 1 : Dataset composition. Museum-65 covers a wide range of exhibit categories (top), e.g arts, historical/pre-historical, natural sciences, and contains a large number of images from around the globe . Each image is paired with multiple questions exploring subjects like Title, Creator, Period, Techniques, Culture, Inscriptions, etc. (right). A sample image with a question and answer is shown on the left. In this paper, we aim to develop AI models with a strong understanding of museum artifacts, by addressing the task of visually understanding exhibit images through visual question answering. Vision-Language Models (VLMs) like CLIP Radford et al. 2021 , Gemini Team et al. 2023 , and LLaVA Liu et al. 2023 , have proven highly effective in training on large amounts of noisy image-text data, considerably improving our understanding of visual content through natural language, and bridging the gap between textual annotations and visual data, with broad applicability Wei et al. 2023 ; Wang et al. 2022 ; Cui et al. 2024 ; Zhai et al. 2022 ; Pham et al. 2023 ; Rasheed et al. 2023 ; Meng et al. 2023 ; Lin et al. 2024 ; Ramesh et al. 2021 . However, these models Li et al. 2022c ; Liu et al. 2023 face limitations in contexts like museums, which require a detailed and interdisciplinary understanding of a long tail of objects, and prediction of structured attributes such as age, origin, material, and cultural relevance Becattini et al. 2023 ; Marty and Jones 2008 ; Nishanbaev et al. 2019 . Pre-trained VLMs, known for their robust visual representations, are typically designed for specific vision-language tasks, like object detection Gu et al. 2021 ; Bangalath et al. 2022 ; Zhou et al. 2022d and semantic segmentation Ding et al. 2021 ; Li et al. 2022b ; Zhou et al. 2022a . Yet, more complex, multi-modal tasks often demand advanced capabilities that go beyond standard visual representations, requiring the processing and alignment of information across the visual and textual domains Ju et al. 2022 ; Ni et al. 2022 ; Wang et al. 2021 ; Rasheed et al. 2023 . Visual Question Answering (VQA) is one such multi-modal task that combines visual understanding with natural language processing, like in Antol et al. 2015 ; Singh et al. 2019 ; Zhu et al. 2016 ; Bigham et al. 2010 ; Malinowski et al. 2015 ; Sadeghi et al. 2015 ; Becattini et al. 2023 . In the cultural heritage domain, VQA can play a key role when paired with a proper dataset. Yet, a dataset covering many artifacts and integrating visual and textual data does not exist. Existing similar datasets are mostly focused on art Ruta et al. 2022 ; Wilber et al. 2017 ; Strezoski and Worring 2018 and are often used in tasks like image generation and style transfer Gatys et al. 2016 ; Kwon and Ye 2022 ; Deng et al. 2022 ; Ramesh et al. 2021 without capturing the deeper relationship between an exhibit and its descriptive context. In this work, we collect a novel large-scale dataset Museum-65 with high-quality images and extensive textual information for a wide range of museum artifacts, totaling 65M images and 200M question-answer pairs across multiple languages. We curate the data and use it to fine-tune VLMs, BLIP and LLaVA, to enable a better understanding of museum exhibits across diverse cultural backgrounds. The textual information of Museum-65 reflects the viewpoint of knowledgeable museum experts, ensuring that it provides both depth and breadth for effective AI training. We further design 5 tasks with real-world applicability: general VQA, category-wise VQA, MultiAngle – questions when different angles occur, Visually Unanswerable Questions – more complex questions requiring the use of general knowledge, and MultiLanguage – questions in languages other than the (English) language used for fine-tuning. To foster further research, we will make this dataset publicly available. This contribution aims to facilitate the development of AI models that can handle complex cross-disciplinary questions in a truthful and comprehensive manner, enabling museums to serve as dynamic educational platforms that enrich visitor experience and deepen understanding across diverse cultural, historical, and scientific domains, as we show by fine-tuning BLIP Li et al. 2022c and LLaVA Liu et al. 2023 BLIP aligns images with descriptive text effectively, generating accurate captions that enhance its question-answering capabilities. Still, BLIP’s smaller text encoder/decoder ( BERT-base , 110M params.) limits its ability to handle complex instructions. LLaVA, powered by the larger Llama2-7B LLM, excels in instruction comprehension and vision-language reasoning, making it capable of performing complex tasks. We provide insights into the nuanced and detailed understanding and real-world applications required for museum exhibits, presenting comparisons of the two models on multiple metrics. We show both can handle questions well when answers can be directly derived from visual features. However, for questions requiring the integration of visual features with broader human knowledge, large VLMs which understand instructions better attain higher accuracy, performing the reasoning needed for such inquiries. For instance, they can answer questions that link visual details to historical facts or explain connections to related events or figures not directly depicted. The major contributions of the paper are: • Dataset and fine-tuned models: We introduce a dataset of 65M images and 200M question-answer pairs for museum exhibits suitable to build new vision-language models and to fine-tune existing ones (e.g. BLIP, LLaVA) • Benchmark: We propose 5 tasks derived from our dataset, setting directions for research in real-world AI for cultural heritage, along with the metrics to evaluate them.