Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilities

Nishant Balepur, Dang Nguyen, Dayeon Ki · 2025

Read Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilities by Nishant Balepur, Dang Nguyen, Dayeon Ki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19892.txt.

Introduction

Nishant Balepur, Dang Nguyen, and Dayeon Ki propose judging multimodal language models by making them play Dixit, the picture-card game of inventive captions. In Dixit a player writes a clue meant to fool some opponents but not all, demanding creativity, social reading, and visual understanding at once, under fixed objective rules. Static benchmarks test skills separately, and pairwise judging is subjective and gameable.

Five multimodal models played the game, and their win-rate rankings matched their rankings on popular benchmarks exactly, validating the method. Games against humans then exposed differences in strategy and concrete weaknesses in model reasoning. The authors argue game-based evaluation gives a holistic, harder-to-exploit picture of what these models can do.

Catalog record

CREATORNishant Balepur, Dang Nguyen, Dayeon Ki
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19892

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilities Nishant Balepur, Dang Nguyen, Dayeon Ki Abstract Multi-modal large language models ( MLMs ) are often assessed on static, individual benchmarks—which cannot jointly assess MLM capabilities in a single task—or rely on human or model pairwise comparisons—which is highly subjective, expensive, and allows models to exploit superficial shortcuts (e.g., verbosity) to inflate their win-rates. To overcome these issues, we propose game-based evaluations to holistically assess MLM capabilities. Games require multiple abilities for players to win, are inherently competitive, and are governed by fix, objective rules, and makes evaluation more engaging, providing a robust framework to address the aforementioned challenges. We manifest this evaluation specifically through Dixit, a fantasy card game where players must generate captions for a card that trick some, but not all players, into selecting the played card. Our quantitative experiments with five MLMs show Dixit win-rate rankings are perfectly correlated with those on popular MLM benchmarks, while games between human and MLM players in Dixit reveal several differences between agent strategies and areas of improvement for MLM reasoning. 1 1 1 https://github.com/nbalepur/dixit 1 Introduction Multi-modal large language models ( MLMs ) have made significant progress, showing impressive performance in tasks requiring both image and text inputs, such as image captioning, classification, and visual understanding ( Zheng et al. 2023b ) . As a testament to this, several MLMs with diverse architectures have been introduced in the past few years, each aiming to excel in such tasks ( Chen et al. 2024b ; Bai et al. 2023 ; Agrawal et al. 2024 ) . As the number of MLMs continues to grow, the need for robust and comprehensive evaluation frameworks becomes increasingly important ( White et al. 2024 ; Saxon et al. 2024 ) . These frameworks are key for reliably comparing models, enabling users to choose those that best align with their needs ( Li et al. 2024 ) . Such MLM benchmarks fall into two main categories. The first approach evaluates models on individual tasks—such as image captioning, classification, or understanding—pinpointing abilities in isolation ( Duan et al. 2024 ; Lee et al. 2024b ) . While useful, this precludes jointly testing model capabilities allowing for direct comparisons between models. The second involves sampling outputs from two models and using pairwise comparison judgments from humans or other models to identify which model is better (i.e. win-rate) ( Wu et al. 2023 ; Chiang et al. 2024 ) . While this better compares models, such evaluations are inherently subjective Lopez et al. 2021 , as they rely on external human or model judgments without objective rules ( Zheng et al. 2023a ) , and such judges are prone to biases like verbosity and position ( Wang et al. 2023 ; Shi et al. 2024 ; Ye et al. 2024 ) . They are also expensive, requiring large-scale human annotation or computational resources for model-based judging ( Lee et al. 2024a ; Chen et al. 2024a ) . Figure 1: Overview of playing Dixit with MLMs. Step 1 : The story-teller model generates a caption “ Freedom’s key ” for one of its cards, which is shown to all players. Step 2 : Each player model select a card from their hand that they think corresponds to the caption. Step 3 : From the pool of selected cards, all player models select the card they believe was the story-teller’s. To address these issues, we draw from Natural Language Processing (NLP) research showing multi-agent LLM evaluations can be conducted through games ( Xu et al. 2024 ; Ye et al. 2024 ; Hu et al. 2024 ) , and propose game-based evaluations as a framework for evaluating MLMs . Games inherently involve competition ( Lebed 2006 ) , enabling direct comparison of model capabilities, with rankings serving as a proxy for model performance. Further, games are governed by fixed rules ( DeVries 2015 ) , eliminating the need for external judges and ensuring objective, efficient evaluations. The rules of games are also designed in a manner to prevent players, in our case models, from exploiting superficial shortcuts to inflate their win-rates ( Webb and Soh 2007 ) . As an added benefit, games are designed to be fun ( Prensky 2001 ) , so practitioners can directly intervene in game-based evaluations to assess their models in a more engaging manner. To advocate for this approach, we adopt Dixit ( Kunda and Rabkina 2020a ; Hsu et al. 2019 ) —a fantasy card game—as our game-based evaluation suite for MLMs , as it evaluates several diverse MLM capabilities (Figure 1 ). In Dixit, one player called the storyteller selects a card from their hand and generates a caption for it. The other players then choose cards from their hands that they believe best match the caption. All of these cards are pooled, and players vote on which card (not their own) they think is the storyteller’s. Non-story teller players score points if they select the correct card, but the storyteller faces a unique challenge to score points: their caption must be sufficiently relevant to guide other players to the correct card but not so obvious such that all players guess correctly; the storyteller does not score points if either no one or everyone guesses their card. Dixit provides a unified and nuanced evaluation suite for multiple MLM capabilities. A strong player must be adept at: 1) generating creative captions that allow only some users to pick the correct card (testing creativity, image comprehension, and calibration in image captioning); and 2) identifying cards that align with what players believe the storyteller to have played and what players believe other players may select (testing accuracy, image comprehension, and theory-of-mind in image classification). Dixit captures diverse model abilities in a single, cohesive task, making it an appealing game for our analysis. We build our own evaluation interface for Dixit and have five MLMs compete as players: GPT-4o ( Achiam et al. 2023 ) , Claude-3.5 Sonnet ( Anthropic 2023 ) , Intern-VL2 ( Chen et al. 2024b ) , Qwen2VL ( Bai et al. 2023 ) , and Molmo ( Deitke et al. 2024 ) . We find that all players significantly outperform a player who randomly generates captions and selects cards, showing that MLMs that are not directly trained on Dixit can still demonstrate above-random Dixit capabilities. Further, ranking players by Dixit performance perfectly matches the rankings on popular leaderboards like ChatbotArena ( Chiang et al. 2024 ) and the Open VLM Leaderboard ( Duan et al. 2024 ) , suggesting that Dixit comprehensively tests MLM capabilities within a single task. We then have humans play against one MLM to study if MLMs can surpass humans in Dixit. GPT-4o Mini, a relatively weak MLM , was able to surpass one human player, showing that MLMs possess strong Dixit capabilities that rival and may even surpass human abilities. Finally, we assess the game logs between MLM players in Dixit to understand differences between model and human Dixit strategies; we find when humans are story-tellers, they tend to generate more abstract captions that reference external knowledge, while MLM captions

Read the complete text