Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; xxx; 2025 ISBN: 978-1-4503-XXXX-X/18/06

Weichang Wu 1 , Xiaolu Zhang 1 , Jun Zhou 1 , Yuchen Li 2 , Wenwen Xia 3 · 2025

Read Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; xxx; 2025 ISBN: 978-1-4503-XXXX-X/18/06 by Weichang Wu 1 , Xiaolu Zhang 1 , Jun Zhou 1 , Yuchen Li 2 , Wenwen Xia 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11053.txt.

Introduction

Weichang Wu and co-authors present a new pre training method for user behavior sequences, the click and action trails behind apps. Their paper from 2025 deals with a core step in past methods: a hand built list of key behaviors to predict. That list is slow to make, open to bias, and capped in size.

Their method predicts a single summary embedding of all behavior in a future time span, built on its own with no hand list. A student and teacher encoder pair builds that target in a sound way. Tests on two real industry data sets and eight later tasks showed gains in accuracy and a near doubling of training speed. A live deployment in a payment app lifted risk scores over two months.

Catalog record

CREATORWeichang Wu 1 , Xiaolu Zhang 1 , Jun Zhou 1 , Yuchen Li 2 , Wenwen Xia 3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11053

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; xxx; 2025 ISBN: 978-1-4503-XXXX-X/18/06 Weichang Wu 1 , Xiaolu Zhang 1 , Jun Zhou 1 , Yuchen Li 2 , Wenwen Xia 3 Abstract . User Behavior Sequence (UBS) modeling is crucial in industrial applications. As data scale and task diversity grow, UBS pretraining methods have become increasingly pivotal. State-of-the-art UBS pretraining methods rely on predicting behavior distributions. The key step in these methods is constructing a selected behavior vocabulary. However, this manual step is labor-intensive and prone to bias. The limitation of vocabulary capacity also directly affects models’ generalization ability. In this paper, we introduce Bootstrapping Your Behavior ( BYB ), a novel UBS pretraining strategy that predicts an automatically constructed supervision embedding summarizing all behaviors’ information within a future time window, eliminating the manual behavior vocabulary selection. In implementation, we incorporate a student-teacher encoder scheme to construct the pretraining supervision effectively. Experiments on two real-world industrial datasets and eight downstream tasks demonstrate that BYB achieves an average improvement of 3.9% in AUC and 98.9% in training throughput. Notably, the model exhibits meaningful attention patterns and cluster representations during pretraining without any label supervision. In our online deployment over two months, the pretrained model improves the KS by about 2.7% and 7.1% over the baseline model for two financial overdue risk prediction tasks in the Alipay mobile application, which reduces bad debt risk by millions of dollars for Ant group. 1. Introduction A user behavior sequence (UBS) captures user actions over time in contexts such as mobile apps or online e-commerce platforms. UBS is fundamental to a variety of downstream applications, such as user personas construction, recommendation systems, and financial risk management ( Chen et al. 2019 ; Benevenuto et al. 2009 ; Zhang et al. 2016 ; Zhu et al. 2020 ; Liu et al. 2021 ) . Traditional UBS models are generally trained using supervised learning, relying on predefined labels for each task ( Pi et al. 2019 ; Chen et al. 2018 ) . As the industry’s data volume and number of tasks increase, pretraining methods for UBS data become increasingly important to support diverse tasks promptly and effectively. For example, various financial products like Huabei and Jiebei in Ant Group involve numerous overdue risk prediction tasks. As these products continue to evolve, having a pre-trained model capable of simultaneously handling these diverse and interrelated tasks is essential for minimizing bad debt risks. Pretraining has emerged as a powerful technique for natural language processing (NLP) in recent years ( Brown et al. 2020 ; Achiam et al. 2023 ; Team et al. 2023 ) . Inspired by the success of pretraining in NLP, many UBS pretraining methods have adopted similar approaches, by either predicting the next behavior ( Wu et al. 2020 ) (see Fig. 1 (a) ), or masked behaviors ( Shin et al. 2021 ; Wu et al. 2022 ; Zhao 2022 ) (see Fig. 1 (b) ). However, directly applying pretraining strategies from the NLP domain to UBS data does not yield satisfactory performance on downstream tasks, because UBS has unique characteristics—such as local randomness and global orderliness—that differ fundamentally from the structured nature of language. For example, in a mobile financial app, a user might switch between stock and fund sections in any order, with no significant impact on the sequence representation (local randomness). Over a longer period, user behavior might shift from higher-risk activities (stocks) to lower-risk ones (fixed deposits), demonstrating long-term trends (global orderliness). Hence, recent works for UBS pretraining aim to predict user behavior distributions ( Fu et al. 2023 ; Gu et al. 2021 ) , i.e., pre-defining a behavior vocabulary and predicting whether each behavior in the vocabulary will happen in a future time window (see Fig. 1 (c) ). Because user behavior distributions tend to be more stable and informative, pretraining models on this strategy lead to state-of-the-art (SOTA) performance. However, a crucial limitation still exists for these methods, i.e., limited behavior vocabulary. SOTA UBS pretraining methods rely on a manually selected subset of behaviors (limited behavior vocabulary). The selected behavior vocabulary must be relevant to downstream tasks. Because the model only focuses on these selected behaviors in pretraining, the quality of behavior vocabulary directly affects model performance. This manual step can be both labor-intensive and prone to human bias. Moreover, the limited vocabulary often overlooks long-tail behaviors, which are crucial for model generalization. Expanding the vocabulary to include more behaviors is computationally expensive and often impractical, as behavior space can be vast—ranging from tens of thousands to millions of unique behaviors ( Zhong et al. 2015 ; Cheng et al. 2021 ) . Figure 1 . Conceptual comparison between our pretraining strategy and previous pretraining strategies for UBS data. To this end, in this paper, we propose a novel pretraining strategy for UBS data, i.e., B ootstrapping Y our B ehavior ( BYB ), and a practical implementation, to address the crucial challenge of behavior vocabulary in SOTA UBS pretraining methods. Pretraining Strategy. Unlike existing methods that predict individual behaviors based on a limited vocabulary, BYB eliminates the need for manually curated behavior vocabularies by generating a supervision embedding that summarizes all behaviors within a future time window(see Fig. 1 (d) ). This approach not only removes the dependency on manual behavior vocabularies but also captures both frequent and long-tail behaviors smoothly. Practical Implementation. In industrial applications, behaviors are typically represented by discrete integer IDs (e.g., 1 for ’stocks’, 2 for ’funds’) rather than semantically rich embeddings. While previous UBS pretraining methods predict the occurrence of behaviors—making it straightforward to construct discrete pretraining labels from raw IDs— BYB faces the challenge of generating an effective embedding as supervision from these raw IDs that encapsulates all behaviors’ information in a future time window. To address this, we introduce a student-teacher behavior encoder scheme, as illustrated in Fig. 2 . The student encoder converts raw behaviors in the observation window [0,T] into embeddings for sequence representation extraction, while the teacher encoder processes behaviors in the prediction window [T,T+\Delta T_{2}] to construct the supervision embedding. The pretraining objective is to predict this supervision embedding using a predictor module, with the extracted sequence representation being applied in downstream tasks post-pretraining. This innovative approach overcomes the limitations of traditional methods and provides a robust framework for UBS pretraining in real-world industrial applications. We summarize our contributions in this paper as follows: • We propose a novel pretraining strategy for general UBS data (Sec. 4.1 ) that eliminates the labor-intensive behavior vocabulary construction common in SOTA methods. • We implement this strategy in BYB (Sec. 4.2 - Sec. 4.3 ) by incorporating the innovative student-teacher encoder design, specifically designed for industrial UBS data with a focus on both efficiency and effectiveness. • We conduct extensive experiments in offline datasets. BYB improves AUC by 3.9% over SOTA baselines and boosts training throughput by 98.9% on two industrial datasets with eight tasks (Sec. 5.1 - Sec. 5.3 ). We also conducted a thorough analysis of the model’s intrinsic patterns and extracted UBS embeddings (Sec. 5.4

Read the complete text