Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning
Yuti Liu, Shice Liu, Junyuan Gao, Pengtao Jiang, Hao Zhang, Jinwei Chen, Bo Li · 2024
Read Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning by Yuti Liu, Shice Liu, Junyuan Gao, Pengtao Jiang, Hao Zhang, Jinwei Chen, Bo Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11952.txt.
Introduction
Yuti Liu and colleagues trained a multimodal model to judge image beauty in depth in a 2024 paper. Aesthetic assessment should score a photo, explain its strengths, and suggest fixes, but models usually train for one such task on scarce labeled data. Their system learns from masses of unlabeled images instead, guided by text at several scales, from whole-image impressions down to local details.
A multi-scale alignment module ties image features to those layered descriptions during self-supervised pretraining. After extensive instruction tuning, the model set new state-of-the-art results on scoring, commenting, and personalized assessment benchmarks. Learning aesthetics without labels at scale may finally give photo tools a critic that explains itself.
Catalog record
| CREATOR | Yuti Liu, Shice Liu, Junyuan Gao, Pengtao Jiang, Hao Zhang, Jinwei Chen, Bo Li |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11952
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning Yuti Liu, Shice Liu, Junyuan Gao, Pengtao Jiang, Hao Zhang, Jinwei Chen, Bo Li Abstract Image Aesthetic Assessment (IAA) is a vital and intricate task that entails analyzing and assessing an image’s aesthetic values, and identifying its highlights and areas for improvement. Traditional methods of IAA often concentrate on a single aesthetic task and suffer from inadequate labeled datasets, thus impairing in-depth aesthetic comprehension. Despite efforts to overcome this challenge through the application of Multi-modal Large Language Models (MLLMs), such models remain underdeveloped for IAA purposes. To address this, we propose a comprehensive aesthetic MLLM capable of nuanced aesthetic insight. Central to our approach is an innovative multi-scale text-guided self-supervised learning technique. This technique features a multi-scale feature alignment module and capitalizes on a wealth of unlabeled data in a self-supervised manner to structurally and functionally enhance aesthetic ability. The empirical evidence indicates that accompanied with extensive instruct-tuning, our model sets new state-of-the-art benchmarks across multiple tasks, including aesthetic scoring, aesthetic commenting, and personalized image aesthetic assessment. Remarkably, it also demonstrates zero-shot learning capabilities in the emerging task of aesthetic suggesting. Furthermore, for personalized image aesthetic assessment, we harness the potential of in-context learning and showcase its inherent advantages. These authors contributed equally. These authors contributed equally. vivo Mobile Communication Co., Ltd Shanghai, China {kira66arik, junyuangao577}@gmail.com; {liushice, pt.jiang, haozhang, jinwei.chen, libra † }@vivo.com Figure 1: The functional comparison of our proposed CALM and other IAA methods. 1 Introduction As artificial intelligence evolves, there’s a growing demand for agents to mimic human perception and exhibit emotional responses to their surroundings. IAA emerges as a key area within this scope, and gauges images’ aesthetic appeal akin to human judgment. Its complexity lies in its subjectivity, governed by factors like photographic subjects and personal experiences, which makes IAA a challenging endeavor. In the last decade, IAA has been concretized into a variety of tasks. EAT ( He et al. 2023b ) predicts aesthetics based on a single human-assigned score per image—a task known as Aesthetic Scoring (AS). Meanwhile, CWS ( Ghosal et al. 2019 ) assesses an image’s aesthetic appeal directly through language, which is referred to Aesthetic Commenting (AC). Recently, Personalized Image Aesthetic Assessment (PIAA) has emerged as a burgeoning field, which aims to predict an individual’s aesthetic preferences based on his historical image scoring. While effective in certain scenarios, approaches focusing on a single task often fail to address linkages between different tasks, suffering from overfitting to specific tasks. This realization has inspired us to prioritize holistic aesthetic analysis and comprehension in our research efforts. Recently, MLLMs have demonstrated strong comprehension and reasoning abilities across various domains. Models such as VILA ( Ke et al. 2023 ) , Q-Align ( Wu et al. 2023 ) , and UNIAA ( Zhou et al. 2024 ) have also attempted to utilize MLLMs for IAA to compensate for perceptual and reasoning processes. However, two major obstacles limit their effectiveness. First, these models rely solely on semantic features, neglecting a wealth of valuable aesthetic information. Second, despite efforts by Q-Align and UNIAA to construct aesthetic question-answer pairs for enhancement, the scarcity of labeled data and the presence of potentially mislabeled data continue to restrict their performance. Consequently, integrating comprehensive aesthetic information into MLLMs and developing a refined learning strategy to accurately leverage massive image data are essential. In this paper, we propose Comprehensive Aesthetic Large language Model (CALM) which excels in various IAA tasks and demonstrates deep aesthetic comprehension and analytical skills in dialogues. Fig. 1 illustrates the functional differences between CALM and other IAA models. Inspired by popular MLLMs, CALM incorporates a visual encoder, a Multi-scale Feature Alignment Module (MFAM) and a Large Language Model (LLM). Recognizing that mainstream visual encoders and LLMs excel at feature extraction and language expression, we have focused our efforts on the MFAM to ensure that the subsequent LLM can fully leverage a broader spectrum of aesthetic information provided by the visual encoder. To achieve this, we introduce a multi-scale text-guided self-supervised learning technique. Specifically, the MFAM is designed to structurally access aesthetic features at multiple levels, while text-guided self-supervised learning enables the MFAM to benefit from unlabeled data. Unlike previous aesthetic self-supervised approaches that rely on score pseudo-labels, our method uses attribute-related textual pseudo-labels. This change ensures accurate learning and simplifies the integration of pseudo-labels when superimposing multiple augmentations on a single image. Additionally, we utilize a wider range of image augmentations, from low-level to high-level, to guarantee that more aesthetic elements are captured and learned. To enhance holistic aesthetic insight, we developed various instruct-tuning techniques to adapt CALM to common aesthetic tasks, ultimately outperforming other approaches in AS, AC, and PIAA tasks. Moreover, CALM achieves comparable PIAA results through in-context learning at runtime, establishing a new paradigm for PIAA. Additionally, we are the first to define the aesthetic suggesting task, and CALM’s zero-shot success in this task demonstrates its ability to grasp and comprehend aesthetic principles effectively. The contributions of our work are concluded as follows: \diamondsuit We propose CALM, a cutting-edge multi-modal large language model specialized in comprehending image aesthetics. Our extensive experiments demonstrate that CALM sets a new benchmark for AS, AC, and PIAA tasks. \diamondsuit We have pioneered a multi-scale text-guided self-sup-ervised learning technique that not only ensures multi-scale perception for MLLMs, but also effectively and efficiently leverages abundant unlabeled images for enhancement. \diamondsuit The remarkable zero-shot capabilities of CALM are explored, particularly in in-context PIAA and providing aesthetic suggestions. These capabilities demonstrate CALM’s comprehensive aesthetic insight and analytical prowess. 2 Related Work Image Aesthetic Assessment involves algorithms that measure the visual appeal of images. Initially, convolutional neural networks (CNN) and transformers have been leveraged to refine aesthetic score predictions, such as TANet ( He et al. 2022 ) , ResNext ( Hou et al. 2022b ) , DAT ( Xia et al. 2022 ) and MaxViT ( Tu et al. 2022 ) . In order to regulate aesthetic features to refine scoring, Comm ( Niu et al. 2022 ) and AesCLIP ( Sheng et al. 2023 ) harness textual data and CLIP ( Radford et al. 2021 ) , respectively. Besides, language generation models for AC task have also emerged, such as Yeo ( Yeo et al. 2021 ) . Moreover, realizing the importance of personal tastes, models and the FLICKR-AES dataset ( Ren et al. 2017 ) for PIAA are gaining traction. However, previous methods usually concentrate on a single aesthetic task so that they can barely really understand aesthetics. Figure 2: The proposed CALM includes a visual encoder, a multi-scale feature alignment module and a large language model. Multi-modal Large Language Models achieve image content analysis by integrating visual features in LLMs. LLaVA-1.5 ( Liu et al. 2023 ) and mPLUG-Owl2 ( Ye et al. 2024 ) have showcased impressive image reasoning skills. In the realm of IAA, VILA employs CoCa ( Yu et al. 2022 ) to explore zero-shot aesthetic judgement, while Q-ALIGN directly utilizes the original mPLUG-Owl2. UNIAA leverages ChatGPT to generate comments to fine-tune LLaVA-1.5. However, these methods do not modify the pre-existing MLLMs and rely on a limited number of constructed data, which may prevent a comprehensive aesthetic