Fine-grained User Behavior Simulation on Social Media Based on Role-playing Large Language Models

Kun Li, Chenwei Dai, Wei Zhou, Songlin Hu · 2024

Read Fine-grained User Behavior Simulation on Social Media Based on Role-playing Large Language Models by Kun Li, Chenwei Dai, Wei Zhou, Songlin Hu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03148.txt.

Introduction

Kun Li, Chenwei Dai, Wei Zhou, and Songlin Hu present a 2024 study on how well large language models can copy the way real people act on social media. Strong role play is not the same as true behavior copying. A model needs to read a person's past posts and act as that person would.

The team built a data set from the full posting history of 1,866 users on three platforms. Each act was split into object, type, and content, giving a large set of test questions. They found two main patterns in how the models reason through a copy task. From those patterns they built a fine-tuning method that made the copies better. Tests show their method beat plain approaches. The work gives builders a firmer way to judge fake-user tools.

Catalog record

CREATORKun Li, Chenwei Dai, Wei Zhou, Songlin Hu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03148

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fine-grained User Behavior Simulation on Social Media Based on Role-playing Large Language Models Kun Li, Chenwei Dai, Wei Zhou, Songlin Hu Abstract Large language models (LLMs) have demonstrated impressive capabilities in role-playing tasks. However, there is limited research on whether LLMs can accurately simulate user behavior in real-world scenarios, such as social media. This requires models to effectively analyze a user’s history and simulate their role. In this paper, we introduce FineRob , a novel fine-grained behavior simulation dataset. We collect the complete behavioral history of 1,866 distinct users across three social media platforms. Each behavior is decomposed into three fine-grained elements: object, type, and content, resulting in 78.6k QA records. Based on FineRob, we identify two dominant reasoning patterns in LLMs’ behavior simulation processes and propose the OM-CoT fine-tuning method to enhance the capability. Through comprehensive experiments, we conduct an in-depth analysis of key factors of behavior simulation and also demonstrate the effectiveness of OM-CoT approach 1 1 1 Code and dataset are available at https://github.com/linkseed18612254945/FineRob . 1 Introduction Large language models (LLMs) have attracted significant attention for their ability to engage in role-playing. These models can be guided by predefined role profiles to generate conversations that align with a character’s speaking style [ 32 ] , knowledge [ 16 ] , and personality traits [ 5 ] . Recently, numerous agent frameworks have been introduced to extend LLMs’ capabilities beyond simple dialogue generation [ 24 , 1 , 23 ] . However, accurately simulating human-like behaviors poses a substantial challenge, particularly in complex real-world scenarios. In this paper, we focus on simulating real social media users’ behavior. Researches have shown that the primary motivations for social network users are self-presentation and self-disclosure [ 20 ] . These self-presentations are consciously or unconsciously reflected in all aspects of user behavior [ 13 ] .Considering that, we are going to break down each behavior into three fine-grained aspects: object (the target or recipient of the behavior), type (the nature of the behavior), and content (the specific details). An ideal role-playing agent should be able to accurately predict each aspect of user behavior. Figure 1: An example of FineRob, requires LLM to simulat behavior choices that align with a role’s profile and historical data. We decompose a complete behavior record into three fine-grained components: selecting the recipient of the action, determining the action type, and specifying the behavior details. Specifically, we introduce the FineRob ( Fine -Grained Ro le B ehavior) dataset, a novel benchmark for role-behavior simulation on social media. FineRob collects real user behavior data from three major platforms—Twitter, Reddit (primarily English), and Zhihu (primarily Chinese)—encompassing 1,866 distinct users and 78.6k fine-grained behavior QAs. Each raw user behavior is decomposed into three aspects. For example, when a user comments on a post, he first chooses a target (e.g., a post or article from their feed), then decides on the type of action (e.g., comment, like, or share), and finally generates content that aligns with his persona, as illustrated in Figure 1 . Using the FineRob dataset, we conduct a comprehensive evaluation of nine widely used LLMs. We analyze the reasoning processes of LLMs and find two prevalent reasoning patterns: " role stereotype-based reasoning " and " observation and memory-based reasoning ". The former tends to over-rely on user profile information, which leads to reduced accuracy in behavior simulation. In contrast, the latter compares current observations with past behaviors, resulting in more accurate simulations, which is favored by more advanced models, such as GPT-4o. Building on this insight, we introduce a novel fine-tuning approach, OM-CoT, which utilizes special tokens to explicitly incorporate observation and memory analysis into the reasoning process. Extended experiments confirm the effectiveness of our approach. The contributions of this study can be summarized as follows: • We introduce the FineRob dataset , consisting of 78.6k fine-grained behavior element simulation QA records from 1866 real-world social media users. FineRob serves as a realistic, multilingual benchmark for evaluating LLMs’ ability to simulate role-specific behaviors. • We comprehensively assess behavior simulation across nine mainstream LLMs, focusing on the reasoning patterns employed during simulation. • We propose the OM-CoT, a novel fine-tuning method that explicitly integrates observation and memory analysis into the reasoning process using special tokens. Our experiments demonstrate significant improvements across all three behavior element simulation tasks. Table 1: Comparison between FineRob and previous role-playing datasets. Dataset Source Size Usage Language Type isReal? Fine-Grained PersonalDialog [ 31 ] Weibo 20.8M Train&Test ZH Dialogue \checkmark – Ditto [ 16 ] LLM Synthetic 4k Train&Test EN,ZH Dialogue \times – LaMP-7 [ 19 ] Twitter 12k Train&Test EN Dialogue \checkmark – PIPPA [ 10 ] Character.AI 26k Train EN Dialogue \times – ROCStories [ 17 ] Little Stories 98K Train&Test EN Behavior \times \times choices13K [ 3 ] gamble 13K Test EN Behavior \checkmark \times Life-Choice [ 25 ] Novels 1.4K Test EN Behavior \times \times FineRob SocialMedias 78.6K Train&Test EN,ZH Behavior \checkmark \checkmark 2 Related Work 2.1 Role-Playing LLM Recently, numerous RP-LLMs have been designed for conversational applications and have already found commercial uses like Character.ai 2 2 2 https://character.ai/ . Researchers collect a wide range of dialogue datasets to support the study, including the data from real-life individuals [ 9 , 6 ] or fictional characters from novels [ 5 , 2 ] . Beyond this, techniques such as in-context learning (ICL) [ 29 ] and retrieval-augmented generation (RAG) [ 15 ] have been employed. Additionally, supervised fine-tuning on targeted dialogue datasets [ 21 ] and methods like LoRA [ 28 ] have further enhanced RP-LLMs’ role-playing capabilities. Despite these advancements, research in RP-LLMs remains in its early stages, with a primary focus on mimicking conversations. Figure 2: Overview of our work, The left and middle sections of the figure illustrate the process of constructing the FineRob dataset. The right section shows how OM-COT-FineTune training details, including data augmentation, reorganize with special tokens and SFT training. 2.2 LLM for User Behavior simulation Researchers have recognized that LLMs are not only adept at mimicking conversation but also capable of simulating complex behaviors [ 18 , 24 , 4 , 23 ] . For instance, [ 22 ] demonstrated that LLMs can mimic real users’ preferences to movie recommendations. [ 8 ] explored the use of reinforcement learning algorithms to optimize recommendation systems based on user feedback simulated by LLMs. The work by [ 25 ] is particularly relevant, which introduced the "LIFECHOICE" dataset to assess LLMs’ ability to make broad, macro-level decisions in fictional contexts. In contrast, our research focus on the LLMs’ capacity to simulate fine-grained, micro-level behaviors in real world scenario, offering a more detailed analysis. 3 FineRob Dataset 3.1 Data Collecting Our goal is to explore how LLMs simulate the behaviors of real internet users. To achieve this, we focused on social media platforms, including Twitter (now X), Reddit, and the Chinese question-and-answer website Zhihu. From these platforms, we collected extensive behavioral histories of real users, as shown in the first part of Figure 2 . Unlike other role-playing tasks, we emphasize fine-grained behavior simulation in real-world scenarios. A detailed comparison is provided in Table 1 . Principles Our data collection strategy is guided by several key principles.

Read the complete text