JaccDiv: A Metric and Benchmark for Quantifying Diversity of Generated Marketing Text in the Music Industry

Anum Afzal, Alexandre Mercier, Florian Matthes · 2025

Read JaccDiv: A Metric and Benchmark for Quantifying Diversity of Generated Marketing Text in the Music Industry by Anum Afzal, Alexandre Mercier, Florian Matthes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20849.txt.

Introduction

Anum Afzal, Alexandre Mercier, and Florian Matthes study machine-made ad text for the music trade, in a paper from two thousand twenty-five. Shops want text made from data, but plain methods tend to repeat the same phrases again and again. A set of near-twin ads is of little use, so range matters as much as quality.

The authors test several large language models, with fine-tuning and with few-shot or zero-shot prompts. They set a base line for text that is both good and varied. They also add a new score, JaccDiv, that gauges how much a set of texts spreads out. The lessons reach past music, to any field where the same kind of text is made at scale.

Catalog record

CREATORAnum Afzal, Alexandre Mercier, Florian Matthes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20849

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JaccDiv: A Metric and Benchmark for Quantifying Diversity of Generated Marketing Text in the Music Industry Anum Afzal, Alexandre Mercier, Florian Matthes Abstract Online platforms are increasingly interested in using Data-to-Text technologies to generate content and help their users. Unfortunately, traditional generative methods often fall into repetitive patterns, resulting in monotonous galleries of texts after only a few iterations. In this paper, we investigate LLM-based data-to-text approaches to automatically generate marketing texts that are of sufficient quality and diverse enough for broad adoption. We leverage Language Models such as T5, GPT-3.5, GPT-4, and LLaMa2 in conjunction with fine-tuning, few-shot, and zero-shot approaches to set a baseline for diverse marketing texts. We also introduce a metric JaccDiv to evaluate the diversity of a set of texts. This research extends its relevance beyond the music industry, proving beneficial in various fields where repetitive automated content generation is prevalent. 1 Introduction In an age where digital transformation is reshaping industries and artificial intelligence is seemingly introduced into every aspect of life, online platforms are adapting their services to ensure a smoother user experience. As most platforms’ attractiveness relies on user-generated content, there is a need to provide users with the tools to create relevant content. LLM-driven technologies can be leveraged to generate text of sufficient quality for mass adoption [ 4 ] , [ 25 ] . However, controlling the quality, richness, and most importantly diversity of generated content is important. Diversity is crucial in the context of marketing on platforms, where competing products and services are often listed side-by-side and where descriptions for said products or services are usually the most distinguishing factor between competitors. In this case, the repetition of similar texts becomes apparent and can lead to a loss of trust in the platform. [ 11 ] show that for the complex domain of humor, ChatGPT repeats variations of the same 25 jokes, which is a high enough number to fool users into thinking that each joke is unique, but probably not diverse enough when marketing descriptions are generated for a platform with possibly hundreds or thousands of competitors. We worked with an online platform aimed at connecting musicians with event organizers. The platform allows the band to create their profile which typically includes a self-description. Since writing about themselves is a time-consuming process, many bands postpone this step or write a short text. To make the band’s profile more appealing, an LLM-driven virtual assistant could help, by generating a description based on the data provided by the band during sign-up. Most studies so far have concentrated on reducing repetitions or controlling text diversity inside a single generated sample. In contrast, we aim to avoid similar structures and increase text diversity between samples. By comparing and contrasting various approaches, we identified the most effective techniques for achieving diverse and engaging marketing content within the music industry context. Our main contributions are as follows: Limited Dataset: As in any industrial use case, the dataset is noisy, limited, and sparse, so many samples do not contain complete band information. We show how to generate good-quality descriptions while leveraging such a dataset. Diversifying Content: Since the input might be similar for many bands, their output is expected to contain repetitions. We experiment with various techniques to increase the diversity of generated text. Text Evaluation: While there are many metrics to evaluate the quality of the generated text, metrics to evaluate diversity are limited. We proposed a new reference-free metric that can be used to measure the diversity between a corpus of texts. 2 Related Work Over the years, many different approaches have been proposed for data-to-text generation. [ 12 ] proposed a zero-shot approach by using multiple pre-trained models such that first the input triplet is formed into facts that are then ordered using a variation of BART-base. RoBERTa is then applied to decide which facts can be aggregated with the last step using BART-base again to aggregate and compress the paragraphs. State-of-the-art approaches however tend to use end-to-end solutions such as GPT models or T5 either in zero-shot mode [ 3 ] or in combination with additional steps such as disambiguation [ 27 ] , reasoning [ 22 ] or Chain-of-Thought prompting [ 26 , 15 ] or even Tree of Thought reasoning [ 28 ] . Although these methods do try to achieve diverse texts, their research focus is concentrated on the quality of the text and controllability [ 29 ] of the models. Some approaches such as [ 8 ] try to tackle this problem by adding relevant words that a model is then supposed to include. Some prompts modification approaches are discussed by [ 20 , 32 ] . For the specific case of question generation, [ 6 ] achieved diverse questions by using multiple selectors that apply a mask on the input data, such that the model can focus on different parts of the input. In the context of product descriptions, tools like Alibaba’s Luban 1 1 1 https://www.alibabacloud.com/luban use AI to generate graphical banners, and a common practice for large international commercial platforms is to use automatic translations in descriptions. Since our research will generate descriptions for users of a platform, the closest topic in marketing is generating descriptions for products in e-commerce [ 30 , 5 ] . Different approaches for diversity evaluation have been tried over the years [ 16 , 23 ] utilizing various embedding techniques to evaluate content diversity. Lastly [ 19 ] attempts to combine quality and diversity measurements into a single score. 3 Dataset We worked with an Industrial dataset acquired from the platform aimed at connecting musicians and event organizers. The dataset contained personal and public information about individual musicians grouped in formations (bands), event organizers who will interact with and hire the formations, and information about the events. We processed this raw dataset to construct a Formation dataset contains the Name, Type, Description, Homebase, Radius (distance they are willing to travel to), Genre and Event types. 3.1 Data Preprocessing & Filtering Since the dataset 2 2 2 The dataset was filtered to remove all the formations without description leaving 440 out of the 880 samples. was in German, the descriptions were translated into English using the DeepL API . A distribution over the number of words in each description of the formation is shown in Figure 1 . The shortest text is only 214 characters long, and the longest is over 3000 characters long. The average length is about 863 characters with a standard variation of 512 characters. Due to some models’ limited attention windows and for comparability, we will try to choose texts of similar lengths in the evaluation set described below. Figure 1: Distribution of description lengths. 3.2 Manual Curation & Quality Control All the remaining descriptions 3 3 3 The dataset was further manually filtered to remove unusable & duplicate descriptions leaving 359 samples. were manually rated in terms of quality and uniqueness. We use these good descriptions as references or templates when generating new descriptions. This subset was used for fine-tuning models and also for few-shot generation. 4 Benchmark 4.1 Models We evaluate five (Large) Language Models in our experiments in terms of Quality and

Read the complete text