Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling Conference: Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining; February 22–26, 2026; Boise, ID, USA Proceedings of the Nineteenth ACM International Conference
Sullivan Castro, Artem Betlei, Thomas Di Martino, Nadir El Manouzi · 2025
Read Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling Conference: Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining; February 22–26, 2026; Boise, ID, USA Proceedings of the Nineteenth ACM International Conference by Sullivan Castro, Artem Betlei, Thomas Di Martino, Nadir El Manouzi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16581.txt.
Introduction
Sullivan Castro and three co-authors pre-train ad models in this 2025 arXiv paper. Display advertising bids in real time on guesses about what a user will do next. Positive events are sparse and erratic, so models starve for signal, and hand-built counter features miss how user intent evolves.
Their method, Abacus, pre-trains a sequential model to predict the empirical frequency distribution of a user's events, teaching it counting statistics that ordinary sequence training skips. A hybrid objective then mixes that task with standard sequential learning, combining the stability of aggregate counts with sensitivity to order. On two real-world data sets, Abacus pre-training beat existing methods and sped up downstream convergence. The full hybrid raised AUC by up to 6.1 percent over baselines, a solid gain for production bidding systems.
Catalog record
| CREATOR | Sullivan Castro, Artem Betlei, Thomas Di Martino, Nadir El Manouzi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-16581
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling Conference: Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining; February 22–26, 2026; Boise, ID, USA Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining (WSDM ’26), February 22–26, 2026, Boise, ID, USA DOI: 10.1145/3773966.3779391 ISBN: 979-8-4007-2292-9/2026/02 CCS: Information Systems Personalization Sullivan Castro, Artem Betlei, Thomas Di Martino, Nadir El Manouzi Abstract . Modeling user purchase behavior is a critical challenge in display advertising systems, necessary for real-time bidding. The difficulty arises from the sparsity of positive user events and the stochasticity of user actions, leading to severe class imbalance and irregular event timing. Predictive systems usually rely on hand-crafted “counter” features, overlooking the fine-grained temporal evolution of user intent. Meanwhile, current sequential models extract direct sequential signal, missing useful event-counting statistics. We enhance deep sequential models with self-supervised pretraining strategies for display advertising. Especially, we introduce Abacus, a novel approach of predicting the empirical frequency distribution of user events. We further propose a hybrid objective unifying Abacus with sequential learning objectives, combining stability of aggregated statistics with the sequence modeling sensitivity. Experiments on two real-world datasets show that Abacus pretraining outperforms existing methods accelerating downstream task convergence, while hybrid approach yields up to +6.1% AUC compared to the baselines. † † cc-license: by 1. Introduction Modeling online user behavior is central to display advertising (DA), where Demand-Side Platforms (DSPs) must decide within milliseconds whether and how much to bid for an impression. The setting is challenging: purchase events are extremely rare relative to clicks and impressions, and event timing is highly irregular, with long and variable gaps between user actions. In practice, production systems rely heavily on handcrafted, count-based features - aggregated statistics over fixed time windows ( 7 ; 28 ; 8 ; 19 ) . These counters are robust and capture coarse patterns, but they under-represent the fine-grained temporal evolution of intent, motivating direct sequence modeling. Deep sequential models have proven their effectiveness in recommendation ( 13 ; 20 ; 29 ) , yet in DA they are often inferior to systems built on counting features, providing stable aggregations ( 6 ; 1 ; 12 ; 9 ) . A common remedy across domains is self-supervised learning (SSL) pretraining before finetuning ( 5 ; 2 ; 3 ) , which brings (i) label efficiency when positives are scarce, and (ii) transferable representations. However, standard SSL tasks (e.g. next item prediction, masked modeling, contrastive learning) do not explicitly align with the counting signals that drive performance. We introduce Abacus , a counting-aligned pretext task that trains a sequential encoder to predict the empirical event-type distribution within a user timeline. Abacus supplies stable supervision that reflects the information captured by production counters (without manual feature engineering) and complements token-level pretexts. We further propose a hybrid objective that mixes Abacus with masked modeling ( 4 ) and Barlow Twins ( 26 ) , combining the robustness of aggregation with the sensitivity of temporal encoders. On a large private dataset, Abacus-based pretraining improves AUC by up to +6.1% over a scratch-trained GRU encoder, with smaller but consistent gains on the public Taobao benchmark. Besides, Abacus yields a warm-start effect and faster convergence. Ablations show that mixing Abacus with complementary pretext tasks is beneficial and stable. Contributions. We (i) show that self-supervised pretraining on event sequences improves user behavior modeling for DA, improving performance and accelerating convergence versus training from scratch, (ii) introduce Abacus , a counting-aligned objective that predicts a sequence’s empirical event distribution (including standard, random, and masking variants) and (iii) propose a hybrid, multi-task approach that mixes Abacus with masking and contrastive pretext tasks, improving accuracy and stability across used encoders. 2. Related Work Early approaches to user modeling for DA relied on LSTM ( 30 ; 14 ) or hybrid ( 11 ) encoders of user activity sequences that captured temporal dependencies in click or browsing logs to make direct predictions. BERT4Rec ( 20 ) , architecture for sequential recommendation, demonstrated combination of bidirectional attention with pretraining via masked modeling to be efficient to model user behavior. Later on, ( 27 ) extended the method by combining masked modeling with sequence rearrangement, explicitly integrating relative time embeddings. In Pretrained User Models (PTUM) ( 24 ) , two pretext tasks for user model pretraining from large-scale unlabeled data are proposed: the model learns relatedness between historical behaviors with masked behavior prediction, while the relatedness between past and future behaviors is captured by next K behavior prediction. UserBERT ( 23 ) complemented PTUM by suggesting pretraining by behavior sequence matching, which aims to capture inherent user interests that are consistent in different periods. ( 15 ) explored benefits of SSL pretraining for event-based user behavior sequences. Their pretraining approach of contrasting the GRU-encoded representations of the possible future events from those of the less likely events with a contrastive loss has demonstrated good performance on the downstream conversion prediction task. ( 17 ) developed PinnerFormer, a Transformer-based model tailored for long user engagement sequences on Pinterest. Unlike methods focusing on next-item prediction, PinnerFormer employs a dense all-action loss that jointly predicts multiple future engagement types (e.g., repins, close-ups, link clicks) across long horizons, demonstrating strong performance in capturing evolving user interests for large-scale recommendation. ( 10 ) proposed taking the prediction of user behavior distribution over a defined period in the future as a pretext task, instead of the usual next K behaviors prediction. They provided the Multi-scale Stochastic Distribution Prediction (MSDP) algorithm, improving the generalization ability of user representation and satisfying various downstream tasks. While the idea of predicting behavior distribution is similar, MSDP aims to extract sequential patterns to predict future actions, differing from our approach of learning present event statistics. ( 16 ) adapted Barlow Twins - contrastive SSL method based on minimizing cross-correlation between pairs of differently corrupted inputs - to user sequence modeling, proven to learn informative representations without relying on labeled data or negative sampling. 3. Method Figure 1 . Illustration of the proposed approach. In pretraining, Abacus, MSM and BT pretext tasks are jointly learned via MTL. Both event and sequence embeddings are then finetuned on downstream task. 3.1. Notation Let the user behavior sequence be x=\{(e_{i},\tau_{i})\}_{i=1}^{\ell} with event type e_{i}\in\{1,\dots,K\} and continuous normalized event timestamp \tau_{i}\in[0,1] , \ell is the sequence length. Pretrained and finetuned model parameters are defined as \theta and \phi respectively. Also, let event embedding be e_{\theta}\in\mathbb{R}^{K\times d} where d is event embedding dimension, sequential encoder be f^{Enc}_{\theta} and sequence embedding be h . Then, pretext task-specific entities are defined: sequence augmentation, encoder projection head, output and loss are a^{t}(x) , {g}^{t}_{\theta} , {o^{t}} and \mathcal{L}^{t} respectively, where t denotes pretext task. For downstream task model, prediction head is denoted as g_{\phi} . 3.2. Pretraining On the pretraining stage, we first apply a task-specific augmentation a^{t} to the raw sequence x , yielding a^{t}(x) . Input representation is a concatenation of the learnable event embedding (e_{\theta})_{i}\in\mathbb{R}^{d} with \tau_{i} , i.e. a^{emb,t}_{\theta}(x)=\big\{\,{(e_{\theta})}_{i}\;\|\;\tau_{i}\,\big\}_{(e,\tau)\in a^{t}(x),i\in\{1,...,\ell\}} . Shared encoder f^{Enc}_{\theta} maps a^{emb,t}_{\theta}(x) to the h=f^{Enc}_{\theta}\big(a^{emb,t}_{\theta}(x)\big) embedding summarizing a sequence. For the encoder architecture, two models are considered: RNN (GRU) and Transformer (BERT with [CLS] token