Demystifying ChatGPT: How It Masters Genre Recognition
Subham Raj, Sriparna Saha, Brijraj Singh, Niranjan Pedanekar · 2025
Read Demystifying ChatGPT: How It Masters Genre Recognition by Subham Raj, Sriparna Saha, Brijraj Singh, Niranjan Pedanekar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03875.txt.
Introduction
Subham Raj, Sriparna Saha, Brijraj Singh, and Niranjan Pedanekar probe how ChatGPT sorts films into genres, in work from twenty twenty-five. Genre tags help streaming tools sort and suggest films to viewers. Past work showed ChatGPT does well at many language tasks. Its skill at genre calls, and its limits there, were not yet clear. The team tests three large language models on the same film task.
The test uses trailer speech text and subtitles from a classic film data set. It covers well over a thousand films across eighteen genres, with multi-tag films allowed. Prompts run with no examples and with a few examples to guide the models. ChatGPT, with no tuning, beats the other models in the plain test. A tuned ChatGPT does best of all in the full runs. Poster images, read by a vision model, add fine detail that lifts the text prompts further.
Catalog record
| CREATOR | Subham Raj, Sriparna Saha, Brijraj Singh, Niranjan Pedanekar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03875
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Demystifying ChatGPT: How It Masters Genre Recognition Subham Raj, Sriparna Saha, Brijraj Singh, Niranjan Pedanekar Abstract The introduction of ChatGPT has garnered significant attention within the NLP community and beyond. Previous studies have demonstrated ChatGPT’s substantial advancements across various downstream NLP tasks, highlighting its adaptability and potential to revolutionize language-related applications. However, its capabilities and limitations in genre prediction remain unclear. This work analyzes three Large Language Models (LLMs) using the MovieLens-100K dataset to assess their genre prediction capabilities. Our findings show that ChatGPT, without fine-tuning, outperformed other LLMs, and fine-tuned ChatGPT performed best overall. We set up zero-shot and few-shot prompts using audio transcripts/subtitles from movie trailers in the MovieLens-100K dataset, covering 1682 movies of 18 genres, where each movie can have multiple genres. Additionally, we extended our study by extracting IMDb movie posters to utilize a Vision Language Model (VLM) with prompts for poster information. This fine-grained information was used to enhance existing LLM prompts. In conclusion, our study reveals ChatGPT’s remarkable genre prediction capabilities, surpassing other language models. The integration of VLM further enhances our findings, showcasing ChatGPT’s potential for content-related applications by incorporating visual information from movie posters. I Introduction In recent years, large language models (LLMs) have emerged as a transformative tool in natural language processing (NLP) research, revolutionizing various fields ranging from machine translation [ 1 , 2 , 3 ] and text generation [ 4 , 5 ] to sentiment analysis [ 6 , 7 ] and information retrieval [ 8 ] . These models, often based on deep learning architectures, possess the capacity to process and comprehend vast amounts of textual data, enabling them to learn complex patterns and relationships within language. Research papers focusing on large language models have delved into several crucial aspects, including model architecture design, training techniques, fine-tuning strategies, and applications across different domains. These studies frequently highlight the challenges associated with model scalability, computational resources, ethical considerations, and the implications of LLMs for society, emphasizing the need for responsible deployment and robust evaluation methodologies. By continually advancing the capabilities of large language models, researchers are actively contributing to the ongoing evolution of artificial intelligence, paving the way for groundbreaking innovations in language understanding and generation. Multi-label movie genre classification is a significant task in the field of machine learning and natural language processing. This task presents unique challenges compared to traditional single-label classification, as movies often belong to multiple genres simultaneously, leading to complex label dependencies and potential ambiguities. Researchers have explored various approaches, including deep learning models such as convolutional neural networks (CNNs) [ 9 , 10 ] , recurrent neural networks (RNNs) [ 11 , 12 ] , and transformer-based architectures like BERT and its variants [ 13 , 14 ] for solving the genre prediction task. In recent studies, VLMs have demonstrated remarkable performance in information extraction from images, as evidenced by the notable works of [ 15 , 16 , 17 , 18 ] . The decision to incorporate VLMs into our research was inspired by the wealth of hidden information embedded in movie posters, which significantly contributes to predicting a movie’s genre. While several deep learning models [ 19 , 20 , 21 ] have leveraged movie posters for multi-label genre classification, an intriguing observation from recent literature highlights the underexplored potential of VLMs in genre prediction tasks. This underscores a unique opportunity for our study to bridge this gap and potentially enhance genre prediction by harnessing the capabilities of VLMs. For our study, we have explored Llava-7B [ 22 ] model 1 1 1 https://huggingface.co/4bit/llava-v1.5-13b-3GB . In this work, we have performed the benchmarking of different LLMs in terms of multi-label movie genre classification. As per our knowledge, we are the first one to perform this benchmarking. We have set up a zero-shot prompt and few-shot prompt in order to trigger LLMs to generate a response from it. We utilized the extracted trailer subtitles from the movies within the MovieLens-100K dataset, employing them as a stimulating prompt. In summary, we have the following major contributions/findings: • Extended the MovieLens-100K dataset by extracting the movie’s trailer’s subtitles and the movie’s poster. • ChatGPT(gpt-3.5-turbo) and its fine-tuned version showed consistent advantages in terms of genre prediction compared to other LLMs like text-davinci-002 and text-davinci-003. Detailed descriptions of these LLMs can be found here . • ChatGPT(gpt-3.5-turbo) under zero-shot setting outperformed traditional classifiers. • We have performed an interesting study for the genre prediction task by integrating llava VLM and LLM. • Cost analysis showed that fine-tuned ChatGPT performed much better compared to non-fine-tuned ChatGPT provided the cost increases, but the few-shot settings performance is at par with the performance of zero-shot settings, hence zero-shot setting is preferred. We hope that this preliminary evaluation of ChatGPT in genre prediction can provide new perspectives on both assessing the capabilities of LLMs and utilizing LLMs, such as ChatGPT, to enhance works such as recommendation systems as these two tasks are closely related [ 23 ] . The remainder of this paper is as follows. Section 2 discusses the dataset. Section 3 discusses the different LLMs and their prompting techniques for genre prediction. Section 4 discusses the experiment and result, which is followed by a conclusion and future works. II Dataset Overview and its Augmentation The MovieLens dataset is a curated collection of movie ratings and user demographic information. It comprises 100,000 movie ratings, represented on a scale from 1 to 5, provided by 943 distinct users for 1,682 movies. It has meta information such as age, gender, occupation, and zip code available for every user, and genre information is available for every movie. In order to set up a prompt for LLMs, we needed the movie’s trailer’s subtitles; therefore, we have extended this dataset by incorporating subtitles for each movie’s trailer present in the MovieLens-100K dataset. Trailer subtitles are extracted via two sources of transcription. Firstly for those movie’s trailer’s whose subtitles were enabled on youtube, we use youtube-transcript-api to transcribe them. For those whose trailer subtitles were disabled on YouTube, we use Youtube-Whisperer to transcribe. The movie’s poster was needed to trigger VLM. In order to collect it, we scraped the IMDb website via BeautifulSoup . This results in the collection of 1682 movie posters present in the dataset. These visuals played a crucial role in expanding our exploration of genre prediction, injecting a cinematic touch to our analysis. III Benchmark Setup III-A Prompting Techniques III-A 1 Zero-shot prompt In our study for the task of movie genre identification, we first employ a zero-shot prompting approach with large language models. The prompt we utilize consists of a comprehensive list of movie genres, denoted as G . Additionally, this prompt incorporates the movie’s trailer’s subtitles, designated as S . Given G in square brackets: G=[ Action, Adventure, Animation, Children’s, Comedy, Crime, Documentary, Drama, Fantasy, Film-Noir, Horror, Musical, Mystery, Romance, Sci-Fi, Thriller, War, Western ] we aim to predict the potential genres G^{\prime} of a movie, represented as G^{\prime}=\textit{LLM}(G,S) (1) where G^{\prime} \subseteq G . Figure 1 (a) shows the LLM zero-shot prompt. III-A 2 Few-shot prompt Under a few show settings, we aim to predict the potential genres G^{\prime} of a movie, represented as G^{\prime}=\textit{LLM}(G,S|S^{\prime}) (2)