Comparison of Different Deep Neural Network Models in the Cultural Heritage Domain

Teodor Boyadzhiev, Gabriele Lagani, Luca Ciampi, Giuseppe Amato, Krassimira Ivanova · 2025

Read Comparison of Different Deep Neural Network Models in the Cultural Heritage Domain by Teodor Boyadzhiev, Gabriele Lagani, Luca Ciampi, Giuseppe Amato, Krassimira Ivanova (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21387.txt.

Introduction

Teodor Boyadzhiev and co-authors present a comparison from two thousand twenty-five of deep learning models for cultural heritage tasks. Computer vision helps document and preserve heritage and enrich visits to museums. Two model families dominate the field: convolutional networks and transformers, which read images in very different ways.

The team tested named models from both families on their power to transfer knowledge from a generic image dataset to heritage-specific tasks. Models included VGG, ResNet, and DenseNet on one side, and several transformer designs on the other. DenseNet gave the best balance of accuracy against computing cost in their tests. The study aims to guide heritage groups that must pick a model under real budget limits.

Catalog record

CREATORTeodor Boyadzhiev, Gabriele Lagani, Luca Ciampi, Giuseppe Amato, Krassimira Ivanova
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21387

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Comparison of Different Deep Neural Network Models in the Cultural Heritage Domain Teodor Boyadzhiev, Gabriele Lagani, Luca Ciampi, Giuseppe Amato, Krassimira Ivanova Abstract The integration of computer vision and deep learning is an essential part of documenting and preserving cultural heritage, as well as improving visitor experiences. In recent years, two deep learning paradigms have been established in the field of computer vision: convolutional neural networks and transformer architectures. The present study aims to make a comparative analysis of some representatives of these two techniques of their ability to transfer knowledge from generic dataset, such as ImageNet, to cultural heritage specific tasks. The results of testing examples of the architectures VGG, ResNet, DenseNet, Visual Transformer, Swin Transformer, and PoolFormer, showed that DenseNet is the best in terms of efficiency-computability ratio. Keywords Deep Learning \cdot Neural Networks \cdot Cultural Heritage 1 Introduction In the dynamic field of computer vision, the recognition of monuments through image analysis has evolved as a compelling application, particularly within the context of cultural heritage Amato et al. 2015 . With the advent of deep learning (DL), there has been a transformative leap in the ability to automatically discern and categorize iconic structures with unprecedented accuracy and efficiency. Indeed, these techniques, such as convolutional neural networks (CNNs) and Transformer, have demonstrated remarkable capabilities in extracting complex features from images, enabling the development of effective models for monument recognition. The integration of computer vision and DL in monument recognition not only facilitates the automated identification of monuments but also contributes significantly to the documentation and safeguarding of cultural heritage Deligiorgi et al. 2021 . By automating the process of cataloguing and classifying monuments, these technologies assist in the creation of comprehensive databases, aiding historians, archaeologists, and conservationists in their endeavours. On the other side, the advent of mobile applications equipped with monument recognition capabilities enhanced visitors’ experience, empowering tourists to actively participate in the classification and documentation of monuments as they explore historical sites Andrianaivo et al. 2019 . In this work, we conduct a comparative experimental analysis of popular DL architectures based on CNNs and Transformers for monument recognition in the cultural heritage domain. Starting from pre-trained publicly available models trained upon extensive image datasets like ImageNet Deng et al. 2009 , our method involves implementing a fine-tuning strategy tailored to the specific task of monument classification. Notably, the capability to adapt to new tasks using only a sparse set of available training samples is critical, as gathering significant amounts of annotated data proves prohibitively expensive. Specifically, we consider VGG-11 Simonyan and Zisserman 2015 , ResNet-34 He et al. 2016 , DenseNet-121 Huang et al. 2017 , ViT-S Dosovitskiy et al. 2021 , Swin-T Liu et al. 2021 , and PoolFormer-S24 Yu et al. 2022 architectures, and we systematically compare their outcomes to monument classification in terms of accuracy and computational cost. We fine-tune and test these models over the Pisa Dataset Amato et al. 2015 , a small collection of 1227 images that includes 12 cultural heritage sites or monuments in Pisa, Italy. The rest of this paper is organized as follows. Section 2 contains a short review of tested architectures. Section 3 describes characteristics of the Pisa dataset. Section 4 is devoted to experimental settings. In Section 5 , the achieved results are analysed. Finally, some conclusions and directions for further research are pointed out in Section 6 . 2 Tested Architectures Past years of research in the domain of computer vision established two essential paradigms for deep learning solutions: Convolutional Neural Network (CNN) and Transformer architectures. The first type of architecture is known to have a relatively low computational footprint, while also being able to generalize easily in image-related problems. This is due to the translation-invariance and locality properties of convolutional layers, which translate to an effective inductive bias, useful for generalization. More recently, Transformer models gained popularity, bringing the ability to capture relationships in the input on a global scale, thanks to their attention-based processing. Unfortunately, the computational complexity of attention layers grows quadratically with the input size, making these models poorly scalable to real-world applications. Some variants of Transformer models have been proposed to address this problem Tay et al. 2023 , replacing full attention blocks with more efficient alternatives, without reducing performance. Transformer-based architectures are known to be less prone to generalization and often require pre-training on very large datasets to achieve satisfactory results Csordás et al. 2021 . Fortunately, pre-trained models of both types of paradigms are publicly available online and can be fine-tuned on task-specific datasets and deployed in a desired applicative scenario. Therefore, in this work, to address the task of monument recognition for cultural heritage applications, we consider some popular architectures belonging both to the category of CNNs, and to that of Transformers, and we conduct a comparative experimental analysis of such models in this scenario. We used a pre-trained model, built on ImageNet and we pursued a fine-tuning strategy on the specific monument classification task, comparing the effectiveness and transferability of such pre-trained features on the new task. Indeed, the capacity to adapt to the new task from few available training samples is crucial, because gathering consistent amounts of annotated data is costly. Table 1: Comparison of different models in terms of number of parameters and computing cost (in GFLOPs). Best values are highlighted in bold. Model # Parameters GFLOPs VGG-11 113M 7.60 ResNet-34 64M 3.68 DenseNet-121 8M 2.91 ViT-S 22M 4.25 Swin-T 28M 4.50 PoolFormer-S24 21M 3.41 Specifically, the models that we consider in our comparison are the following: • VGG-11 Simonyan and Zisserman 2015 is a purely convolutional architecture with 11 layers and 113M parameters; • ResNet-34 He et al. 2016 is an evolution of purely convolutional architectures which introduces residually connected blocks, allowing to stabilize gradient backpropagation and model training even with deeper architectures; it has 34 layers and 64M parameters; • Dense Network – DenseNet Huang et al. 2017 further enhances the residual connection structure by adding skip connections from each layer to every subsequent layer, thus further empowering the representation power and gradient propagation abilities of DenseNet models; in particular, DenseNet-121 uses 121 layers and 8M parameters; • Vision Transformer – ViT Dosovitskiy et al. 2021 encodes images as a collection of tokens, extracted from non-overlapping image patches of fixed size, and then applies a series of Transformer encoder blocks on these tokens, thus leveraging a Multi-Head Self-Attention mechanism to model global relationships among tokens; ViT-S version has 12 layers and 22M parameters; • Swin Transformer Liu et al. 2021 is a state-of-the-art Transformer-based architecture which leverages attentional processing on local windows, whose location is shifted from one layer to the next, to allow information to propagate also along adjacent windows; the Tiny version (Swin-T) of the architecture contains 12 Transformer blocks (implemented leveraging the Swin attention mechanism described so far), and 28M parameters; • PoolFormer Yu et al. 2022 is another state-of-the-art architecture based on the Transformer structure, which instead replaces attention blocks with a simple average pooling, experimentally showing that the performance of the trained models remains comparable to that of purely attention-based models; PoolFormer-S24 version amounts to 24 layers, and 21M parameters. Table 1 summarizes the number of

Read the complete text