FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning
Pulkit Agrawal, Rukma Talwadker, Aditya Pareek, Tridib Mukherjee · 2025
Read FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning by Pulkit Agrawal, Rukma Talwadker, Aditya Pareek, Tridib Mukherjee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21383.txt.
Introduction
Pulkit Agrawal, Rukma Talwadker, Aditya Pareek, and Tridib Mukherjee present FAST-Q, a learning method for game recommendation systems that must improve from logged data alone. Standard offline reinforcement learning plays it safe: it distrusts any action the logs rarely show, which blocks learning about what players might have liked. Player psychology and skewed experiment traffic make the logged states patchy and biased.
FAST-Q attacks the bias directly. A gradient-reversal trick builds state descriptions that hide which policy produced them, letting the system fairly judge untried actions. It can explore those actions offline, in parallel with exploiting the logs. A split value function also separates short-term and long-term goals, so recommendations can be explained. Tests showed gains in player returns and lifetime, small in percentage but real at platform scale.
Catalog record
| CREATOR | Pulkit Agrawal, Rukma Talwadker, Aditya Pareek, Tridib Mukherjee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-21383
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning Pulkit Agrawal, Rukma Talwadker, Aditya Pareek, Tridib Mukherjee Abstract . Recent advancements in state-of-the-art (SOTA) offline reinforcement learning (RL) have primarily focused on addressing function approximation errors, which contribute to the overestimation of Q-values for out-of-distribution actions — a challenge that static datasets exacerbate. However, high-stakes applications such as recommendation systems in online gaming, introduce further complexities due to players’ psychology/ intent driven by gameplay experiences and the platform’s inherent volatility. These factors create highly sparse, partially overlapping state spaces across policies, further influenced by the experiment path selection logic which biases state spaces towards specific policies. Current SOTA methods constrain learning from such offline data by clipping known counterfactual actions as out-of-distribution due to poor generalization across unobserved states. Further aggravating conservative Q-learning and necessitating more online exploration. FAST-Q introduces a novel approach that (1) leverages Gradient Reversal Learning to construct balanced state representations, regularizing the policy-specific bias between the players’ state and action thereby enabling counterfactual estimation; (2) supports offline counterfactual exploration in parallel with static data exploitation; and (3) proposes a Q-value decomposition strategy for multi-objective optimization, facilitating explainable recommendations over short and long-term objectives. These innovations demonstrate superiority of FAST-Q over prior SOTA approaches and demonstrates at least 0.15% increase in player returns, 2% improvement in lifetime value (LTV), 0.4% enhancement in the recommendation driven engagement, 2% improvement in the players’ platform dwell time and an impressive 10% reduction in the costs associated with the recommendation, on our volatile gaming platform. 1. Introduction Offline reinforcement learning (RL), or batch RL, has revolutionized recommender systems by effectively modeling dynamic long-term preferences and short-term stochastic patterns using offline data ( Lange et al., 2012 ; Fujimoto et al., 2019 ) . In skill-based real-money games, such as Rummy ( Wikipedia contributors, 2024 ) , player behavior exhibits significant stochasticity, driven by hidden intent shaped by past gameplay experiences ( Agrawal et al., 2024 ; Talwadker et al., 2022 ) . For instance, a certain player in response to losses might become demotivated and hence dis-engaged, while another could get highly-motivated towards chasing wins - two extremities resulting from the same outcome. Decisions rooted in this organic intent can lead to varied effects: they might reduce immediate engagement but foster long-term retention, conversely, encourage overspending and increase the risk of churn. Our objective is to recommend optimal challenges for each player, balancing immediate and long-term engagement while managing associated costs on our volatile platform. Challenges consist of a specific number of games and target scores, paired with cash rewards upon completion. On our platform, a range of recommendation policies were developed—(i) Policy-1: tailoring player targets based on each player’s past performance; (ii) Policy-2: incorporating the player’s intent to play on a given day ( Agrawal et al., 2024 ) to predict challenges; and (iii) Policy-3: relying solely on the player’s organic engagement, offering no additional rewards beyond the wins on the table (i.e., no specific challenges). None of these policies consistently outperformed each other in our online A/B testing setup and catered to players’ long-term engagement. This motivated us to build FAST-Q which was presented with some unique challenges. 1 1 1 This is the author’s version of the work accepted for publication in the ACM WWW Companion’25. The final version is published by ACM and is available at: https://doi.org/10.1145/3701716.3715224 1. Variable Action Adaptations over Disparate State Spaces: Figure 2 illustrates more disparate and distinct assignment of state spaces across two policy classes on our platform with higher sum of the Wasserstein distance across the two dimensions, compared to the mixed-policy Gym-MuJoCo Hopper task in the D4RL dataset ( Fu et al., 2021 ) used as a representative benchmark. Figure 2 also illustrates higher degree of sparsity in our data, with our data consistently exhibiting a greater number of distinct clusters, indicating more sparse data distribution. Further details on the methods used is in Appendix A.4 . Figure 3 further highlights the disparity in recommended actions across policy classes when provided with identical state representations, showing a higher mean on the difference between actions as compared to the Hopper dataset. Policy switch line in the Figure 3 to be discussed in the Section 4.2.1 Exclusivity in state-action representations stemmed from various factors, including the independent evolution of policy classes and the differing platform priorities shaped by the nature of player engagement across these paths. Figure 1 . Difference in state isolation across policies in Gym-MuJoCo dataset w.r.t Players Data Figure 2 . Higher data sparsity in our Players Data (a) a) Hopper Data (b) b) Player Data Figure 3 . High Variability of actions across policies for identical states on our platform compared to Gym-MujoCo Hopper Task Dataset 2. Normalized State Representation for Q-Value Estimation: Figure 3 indicates that counterfactual actions (actions taken by the alternate policy) are available. However, the state representations being different across the various policies, these actions would be treated as out of distribution samples (OOD) ( Kumar et al., 2020 ; Kostrikov et al., 2021 ; Fujimoto et al., 2018 ; Kumar et al., 2021 ; Fujimoto and Gu, 2024 ; Wang et al., 2023 ) . Figure 4 illustrates spread of Q-values against the actual and counterfactual actions (alternate policy) for a player across states. The exercise is repeated over two competitive SOTA models: TD3+BC ( Fujimoto and Gu, 2024 ) and Diffusion-QL ( Wang et al., 2023 ) . We observe that the optimal policies learned by each of them are invariant to the counterfactual actions resulting in a narrow band of estimated Q-values. Figure 4 . Narrow spread of Q-values against contrastive actions in SOTA models This is further exacerbated by the policy-specific bias towards time-varying confounders and covariates in the data, leading to varying actions (Further discussed in the Section 3.4 ). This bias creates a need for a nuanced approach towards robust generalization of state representations across policies as previously highlighted in ( Fujimoto and Gu, 2024 ; Fujimoto et al., 2019 ) . This bias is well controlled in the Gym-MuJoCo tasks as the states are fully observable. Figure 5 . Cash rewards do not add benefits for players with organic intent to play Figure 6 . Poor generalization of the state space across policies in SOTA 3. Engagement Volatility, Lack of Explainability: Online gaming platforms experience significant fluctuations in player engagement (see Appendix A.3 ). In such businesses where short term revenue is actively tracked, a recommendation system focused on retention should do so with an explanation. The SOTA focuses on maximizing the long term reward with a limited focus on an instantaneous reward as a scalar quantity. Figure 6 depicts that players with high organic intent to play ( Talwadker et al., 2022 ; Agrawal et al., 2024 ) may respond differently to cash rewards than those with low intent. This necessitates variable objective prioritization for a player over time with a specific focus on retention. 4. Fast-Track Offline Counterfactual Exploration: Figure 6 illustrates state representation by Diffusion-QL ( Wang et al., 2023 ) model on Players Data. We observe a lower state isolation as compared to that seen in