PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIG

Chenglong Ma, Ziqi Xu, Yongli Ren, Danula Hettiachchi, Jeffrey Chan · 2025

Read PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIG by Chenglong Ma, Ziqi Xu, Yongli Ren, Danula Hettiachchi, Jeffrey Chan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04551.txt.

Introduction

Chenglong Ma, Ziqi Xu, Yongli Ren, Danula Hettiachchi, and Jeffrey Chan simulate recommender users with personalities, in PUB, a language-model-driven behavior simulator. Their paper from twenty twenty-five faults offline evaluation for sparse signals, noise, and blindness to user character. PUB infers Big Five traits from each user's ratings and reviews, joined with item metadata. The inferred personality then steers synthetic behavior generation.

On Amazon review data, the fabricated logs hug real behavior closely. Trait patterns line up with recommendation outcomes in meaningful ways. Evaluators gain a scalable, controllable stand-in for costly live experiments. Recommender research gets a wind tunnel where the weather, and the shoppers, can be dialed at will.

Catalog record

CREATORChenglong Ma, Ziqi Xu, Yongli Ren, Danula Hettiachchi, Jeffrey Chan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04551

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730238 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Recommender systems Chenglong Ma, Ziqi Xu, Yongli Ren, Danula Hettiachchi, Jeffrey Chan Abstract . Traditional offline evaluation methods for recommender systems struggle to capture the complexity of modern platforms due to sparse behavioural signals, noisy data, and limited modelling of user personality traits. While simulation frameworks can generate synthetic data to address these gaps, existing methods fail to replicate behavioural diversity, limiting their effectiveness. To overcome these challenges, we propose the P ersonality-driven U ser B ehaviour Simulator (PUB), an LLM-based simulation framework that integrates the Big Five personality traits to model personalised user behaviour. PUB dynamically infers user personality from behavioural logs (e.g., ratings, reviews) and item metadata, then generates synthetic interactions that preserve statistical fidelity to real-world data. Experiments on the Amazon review datasets show that logs generated by PUB closely align with real user behaviour and reveal meaningful associations between personality traits and recommendation outcomes. These results highlight the potential of the personality-driven simulator to advance recommender system evaluation, offering scalable, controllable, high-fidelity alternatives to resource-intensive real-world experiments. 1 1 1 The source code can be found at https://github.com/ChenglongMa/PUB . † † cc-license: by 1. Introduction Modern recommender systems (RSs) face critical challenges in evaluation methodologies, particularly as traditional offline datasets struggle to capture the dynamic complexity of user interactions on contemporary platforms. These datasets often lack granular behavioural signals (e.g., personality-driven decision-making) and exhibit biases from sparse or noisy logs, limiting their utility for robust system optimisation. While user studies and real-world datasets provide valuable insights, they are resource-intensive, suffer from uncontrollable confounding variables ( Ma et al. 2022a ; Zhang et al. 2025 ) , and fail to reproduce differentiable evaluation results for modern RSs (e.g., short video platforms) ( Xu et al. 2025 ) . Recent advances in large language models (LLMs) offer promising avenues for simulating user behaviour ( Wang et al. 2024 ; Gao et al. 2024 ) , yet existing frameworks remain inadequate in modelling individual differences, such as personality traits, which significantly influence preferences and engagement patterns. The integration of personality-driven modelling into RSs ( Dhelim et al. 2023 ) has demonstrated benefits, including improved accuracy and cold-start mitigation. The Big Five personality model ( Goldberg 1992 ; Roccas et al. 2002 ) (i.e., Openness (O), Conscientiousness (C), Extraversion (E), Agreeableness (A), Neuroticism (N)) has emerged as a gold standard due to its psychological validity and cross-cultural reproducibility. However, simulating these personality traits at scale remains challenging. Prior work in personality computing primarily focuses on inferring traits from static user data (e.g., social media posts ( Azucar et al. 2018 ; Lambiotte and Kosinski 2014 ) ), while LLM-based simulations often prioritise generic behavioural patterns over trait-specific dynamics. For instance, research ( Sorokovikova et al. 2024 ; Safdari et al. 2023 ; Jiang et al. 2023 ) reveal that LLMs like GPT-4 exhibit distinct Big Five profiles under varying prompts, but their stability and applicability to recommendation tasks remain understudied. Existing simulation tools face two key limitations: (1) Low fidelity : Generated behaviours often deviate from real-world statistical distributions, undermining the reliability of recommender system evaluation ( Ma et al. 2022b ; Gao et al. 2024 ) . (2) Oversimplified personalisation : Collaborative filtering hybrids or Markov decision processes based methods fail to capture the nuanced correlations between user traits and behaviours ( Ie et al. 2019 ; Wang et al. 2024 ) . Despite their potential to mitigate ethical and logistical challenges associated with real-user experiments, such as privacy concerns and experimental costs, these limitations hinder the widespread adoption of simulation tools for large-scale recommender system evaluation ( Wang et al. 2024 ; Gao et al. 2024 ) . To address these challenges, we propose P ersonality-driven U ser B ehaviour Simulator (PUB), which synergises LLM-driven personality inference with dynamic behaviour simulation. Our approach is based on two key insights. First, user interactions encode rich psychological signals that serve as proxies for personality traits ( Tsao and Chang 2010 ; Lambiotte and Kosinski 2014 ; Azucar et al. 2018 ) . PUB leverages these digital footprints along with item metadata to enhance LLM-based trait inference, improving the fidelity of simulated behaviours. Second, recent studies ( Roccas et al. 2002 ; Sorokovikova et al. 2024 ; Jiang et al. 2023 ; Safdari et al. 2023 ) show that LLMs can simulate stable personality profiles when guided by structured prompts. PUB extends this by grounding simulations in empirically validated Big Five correlations, ensuring psychologically coherent behavioural outputs. In summary, our paper makes the following contributions: • Personality-driven simulation : We introduce a personality-based user behaviour simulation framework that dynamically maps inferred Big Five traits to probabilistic behaviour models. • Evaluation robustness : Experiments show PUB-generated logs can effectively replicate performance trends of various recommendation algorithms when compared to real data. • Fine-grained analysis : Our analysis reveals significant correlations between personality traits and recommendation susceptibility, offering new insights into user behaviour dynamics. 2. Methodology Overview of the proposed PUB architecture. Figure 1 . Overview of the proposed PUB architecture. Overview of the proposed PUB architecture. The proposed PUB is a hybrid architecture designed to simulate personality-driven user behaviour for recommender system evaluation. As illustrated in Fig. 1 , the framework operates in four phases: (1) User Profile Aggregator : Extracts user behavioural logs and computes aggregated features (e.g., purchase frequency, category preferences) using statistical functions. (2) Metadata Enhancer : Embeds item metadata (e.g., title, description, price) to enrich contextual information for downstream processing. (3) Personality Inference Module : Maps user behaviour logs and item metadata to Big Five personality traits, leveraging prompt-guided LLMs and psychometric mapping. (4) User Behaviour Simulator : Generates synthetic interactions conditioned on inferred traits to preserve statistical fidelity to real-world patterns in downstream tasks. PUB is designed to be dataset-agnostic and applicable across various domains. In the following sections, we use Amazon review datasets ( Hou et al. 2024 ) as an example. 2.1. User Profile Aggregator The User Profile Aggregator employs a hierarchical approach to construct statistical user profiles from heterogeneous behavioural data. We formalise the process through two primary stages: Behavioural Signature Extraction and Temporal Stratified Sampling. Behavioural Signature Extraction . The core behavioural patterns are formalised through statistical feature engineering. We first aggregate all category-specific interactions into a unified dataset, denoted as \mathcal{D}=\bigcup\left\{\mathcal{D}_{c}\right\}_{c\in C} , where C represents the complete set of Amazon categories. Subsequently, we extract user behavioural logs, comprising sequential interactions d=(i_{k},t_{k},r_{k},c_{k}) , where i_{k} denotes the item ID, t_{k} the timestamp, r_{k} the rating (1–5 stars), and c_{k} the item category. The module computes aggregated features (e.g., purchase frequency, purchase rhythm, category preferences, and price tier distribution) via domain-specific functions. For example, to model purchase rhythm, we employ circular statistics to capture periodic behavioural patterns based on interaction timestamps. Each interaction timestamp t_{i} is mapped onto a unit circle using

Read the complete text