Dialogue with the Machine and Dialogue with the Art World: Evaluating Generative AI for Culturally-Situated Creativity
Rida Qadri, Piotr Mirowski, Aroussiak Gabriellan, Farbod Mehr, Huma Gupta, Pamela Karimi, Remi Denton · 2024
Read Dialogue with the Machine and Dialogue with the Art World: Evaluating Generative AI for Culturally-Situated Creativity by Rida Qadri, Piotr Mirowski, Aroussiak Gabriellan, Farbod Mehr, Huma Gupta, Pamela Karimi, Remi Denton (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14077.txt.
Introduction
Rida Qadri and co-authors (2024) propose judging creative AI by conversation rather than by score. Standard evaluations, benchmarks, crowd-worker ratings, even artist focus groups, miss how art lives inside a social world of curators, historians, and traditions. Borrowing the sociologist Howard Becker's idea of art worlds, the authors run two linked dialogues. In one, artists talk with experts such as art historians and archivists. In the other, artists and critics experiment hands-on with current generative tools.
The case study works with artists rooted in the Persian Gulf's art world, far from the Western data these models mostly absorb. The dialogues trace where the tools help, where they flatten local meaning, and what culturally situated creativity demands that a benchmark cannot see. The method is the contribution: evaluation as an ongoing, two-way conversation between the machine and the world that decides what art is.
Catalog record
| CREATOR | Rida Qadri, Piotr Mirowski, Aroussiak Gabriellan, Farbod Mehr, Huma Gupta, Pamela Karimi, Remi Denton |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14077
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dialogue with the Machine and Dialogue with the Art World: Evaluating Generative AI for Culturally-Situated Creativity Rida Qadri, Piotr Mirowski, Aroussiak Gabriellan, Farbod Mehr, Huma Gupta, Pamela Karimi, Remi Denton Abstract This paper proposes dialogue as a novel and experimental qualitative method for evaluating generative AI tools for culturally-situated creative practice. Drawing on sociologist Howard Becker’s concept of Art Worlds , this method expands the scope of traditional AI and creativity evaluations beyond benchmarks, user studies with crowd-workers, or focus groups conducted with artists. Our method involves two mutually informed dialogues: 1) ‘dialogues with art worlds,’ placing artists in conversation with experts such as art historians, curators, and archivists, and 2) ‘dialogues with the machine,’ facilitated through structured artist- and critic-led experimentation with state-of-the-art generative AI tools. We demonstrate the value of this method through a case study with artists and experts steeped in non-western art worlds, specifically the Persian Gulf. We trace how these dialogues help create culturally rich and situated forms of evaluation for representational possibilities of generative AI that mimic the reception of generative artwork in the broader art ecosystem. They also allow artists to shift their use of the tools to respond to their cultural and creative context. Our study can provide generative AI researchers an understanding of complex dynamics of technology, creativity and the socio-politics of art worlds, to build more inclusive machines for diverse art worlds. 1 Introduction In his seminal work Art Worlds , sociologist Howard Becker emphasized the fundamentally collaborative nature of art creation [ 1 ] . Instead of focusing on a singular, isolated genius or on the aesthetic output, he argued for the study of an ‘art world’ – the network of critics, curators, suppliers, administrators, and audiences who shape artistic activity. The study of art in this conception should not be isolated to artistic outputs, but also encompass the social processes that make art production possible. Said differently, we should study what artists do and how they do it, not just artistic symbolism or composition. Art production in this view is an inherently social process, not an individual or technical one. This scholarship changed the way art was studied [ 2 ] . As generative AI is integrated into the artistic domain, it also enters into existing art worlds and creates the possibility of shaping its own art worlds [ 3 ] . Applying Becker’s perspective to generative AI, the existing evaluation paradigm of AI for creativity must necessarily become ‘social’ and thus expand in both its focus (beyond the output) and in who it involves (beyond the artist). Yet, existing benchmark-centered studies on creativity have focused on quantitative metrics for outputs (e.g., based on human evaluations of the generated image’s quality vs. its prompt such as in DrawBridge [ 4 ] , PartiPrompts [ 5 ] , or conversely, of the ambiguity of crowdsourced annotations of images [ 6 ] ) or on automated methods and datasets for Artistic Image Aesthetic Assessment [ 7 ] , including measures of the diversity of generated images [ 8 ] , of their novelty vs. an existing dataset [ 9 ] or conversely, similarity to an desired image style [ 10 ] . More recently, there has been an emerging focus on socio-technical evaluations of AI. These approaches draw on qualitative methods, bringing in more participatory perspectives, and ultimately expand the frame of what is being evaluated [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ] . While these get closer to understanding AI as a socio-technical artifact, these studies have primarily centered on conversations with artists through interviews, with only a handful studies focusing on artists experimenting with tools directly or bringing artists into conversation with one [ 18 , 19 , 21 , 22 , 23 , 24 ] . Their scope is also often narrowly focused on the usefulness of a generative AI tool for artists as a creativity support tool [ 25 , 26 ] , even as they generate great insight into the individual artist’s interactions with AI and demonstrate the various ways in which artists use and conceptualize emerging AI tools. Other qualitative studies focus on economic impacts for artists [ 27 , 28 , 29 ] and perceptions of creativity and collaboration [ 11 , 12 ] . The scope and method of these qualitative studies could be expanded to explore the ways in which generative art will be received by art worlds, shape new art worlds, or how collectively cultural communities could push the possibilities of representation in generative AI. We fill this gap by introducing an experimental dialogic evaluation of AI for creativity coupling two types of dialogues: dialogue with the machine and dialogue with the art world. These mutually informed dialogues are a step towards a more culturally informed evaluation of AI in art, that can mimic its reception in the art world, internalize its social impacts while also honing in on the particular creative processes it enables or constrains. In this paper we present the application of our approach within a concrete case study where we explored generative AI as a creative tool for cultural representation [ 30 ] , within the cultural context of the Persian Gulf. We instantiated the dialogue with machines as a multi-week experimentation period with artists using state-of-the-art generative AI tools. We instantiated the dialogue with the art world as workshops and reflections with artists and other stakeholders of the collective art world that AI is entering into, such as art historians, curators and archivists. The dialogue was mediated by researchers working in an AI lab who acted as facilitators of the workshops, by asking questions and pairing artists and experts for 1:1 dialogue. The remainder of the paper is structured as follows. First, we introduce our dialogic evaluation method, motivating its value and detailing how we approached the two dialogues within our case study. Next, we present our case study and trace two dialogues that exemplify the value of this approach. Finally, we close by reflecting on our approach and the unique insights offered for Generative AI development through dialogue as a method. As a demonstration of the value of elevating diverse forms of multi-disciplinary expertise for AI evaluations, this paper has been co-authored with scholars and artists who participated in the study. 2 Dialogue as Method In this section we introduce two mutually informed sets of dialogue: dialogue with art worlds and dialogue with machines. Throughout this section, we motivate the value of each dialogue and then describe our approach to instantiating the dialogue within our case study situated within the cultural context of the Persian Gulf. 2.1 Dialogue with Art Worlds: Evaluating Creativity as an Ecosystem According to Martin Zeilinger, human creativity is “dialogic, relational, and fundamentally intertextual” – a social act [ 31 ] . One of the lessons of art history is that art was not developed in vacuum, but has been a dialogue between creator, receivers, audience, about symbols, processes and interpretation [ 32 , 33 , 1 ] . Then, evaluations of art are also necessarily social. We propose a method that allows us to engage