Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment

Mathieu Petitbois, Rémy Portelas, Sylvain Lamprier · 2026

Read Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment by Mathieu Petitbois, Rémy Portelas, Sylvain Lamprier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22823.txt.

Introduction

Mathieu Petitbois, Rémy Portelas, and Sylvain Lamprier study offline learning where style matters as much as success. An agent learns from logged data, without fresh trial and error. The goal is a policy that both performs well and follows a target style, labeled on parts of past behavior.

Style and reward often pull in opposite ways, and offline data shift makes the clash worse. Past methods, despite many definitions of style, often fail to balance the two. Workable style control matters for robots and games that must act well in a set manner.

Catalog record

CREATORMathieu Petitbois, Rémy Portelas, Sylvain Lamprier
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22823

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment Mathieu Petitbois, Rémy Portelas, Sylvain Lamprier Abstract We study offline reinforcement learning of style-conditioned policies using explicit style supervision via subtrajectory labeling functions. In this setting, aligning style with high task performance is particularly challenging due to distribution shift and inherent conflicts between style and reward. Existing methods, despite introducing numerous definitions of style, often fail to reconcile these objectives effectively. To address these challenges, we propose a unified definition of behavior style and instantiate it into a practical framework. Building on this, we introduce Style-Conditioned Implicit Q-Learning (SCIQL), which leverages offline goal-conditioned RL techniques, such as hindsight relabeling and value learning, and combine it with a new Gated Advantage Weighted Regression mechanism to efficiently optimize task performance while preserving style alignment. Experiments demonstrate that SCIQL achieves superior performance on both objectives compared to prior offline methods. Code, datasets and visuals are available in: https://sciql-iclr-2026.github.io/ . 1 1 footnotetext: Correspondence to [email protected] 1 Introduction A task can often be performed through diverse means and approaches. As such, while the majority of the sequential decision making literature has focused on learning agents that seek to optimize task performance, there has been a growing interest in the development of diverse agents that display a variety of behavioral styles, which can be crucial in human-robot interaction, autonomous driving and video games. While many previous works tackled diverse policy learning by relying on online interactions ( 41 ; 58 ) , the widespread availability of pre-recorded diverse behavior data ( 22 ; 36 ; 63 ; 14 ; 28 ; 23 ; 43 ) catalyzed much progress in the learning of policies from such data without further environment interactions, allowing the training of high-performing agents in a more sample-efficient, less time-consuming and safer way ( 30 ) . Such methods can be divided into two categories: Imitation Learning (IL) methods ( 48 ; 13 ; 9 ) mimic expert trajectories, while offline Reinforcement Learning (RL) methods ( 27 ; 26 ; 16 ; 7 ; 39 ; 17 ) target high-performing behaviors based on observed rewards. Although some recent work has focused on diverse policy learning in both offline IL ( 62 ; 59 ) and offline RL ( 38 ) , several challenges remain. Challenge 1: Style definition. Literature dealing with style alignment ranges from discrete trajectory labels ( 62 ; 59 ) to unsupervised clusters ( 38 ) and continuous latent encodings ( 47 ) , with distinct trade-offs: unsupervised definitions are often uncontrollable and hard to interpret, while supervised ones rely on manual labels and incur significant labeling costs. Additionally, since play styles span multiple timescales, attributing each local step to a style is non-trivial and can take part in credit assignment problems. Furthermore, depending on the definition of style, assessing the alignment of an agent’s behavior with respect to a target style may be difficult, which complicates alignment measurement and hinders policy controllability. As such, a key challenge is to derive a general definition that addresses interpretability , labeling cost , alignment measurement , and credit assignment . Challenge 2: Addressing distribution shift. While offline IL and offline RL are known to suffer from distribution shift due to environment stochasticity and compounding errors ( 30 ) , the addition of style conditioning can exacerbate the issue by creating mismatches at inference time between visited states and target styles. For instance, a running policy may trip and fall into an out-of-distribution state-style configuration without the ability to recalibrate. While some previous work addressed this issue ( 47 ) , most of them lack mechanisms to perform robust style alignment. Consequently, an open question is how to achieve robust style alignment without relying on further environment interactions. Challenge 3: Solving task and style misalignment. Style alignment and task performance are often incompatible. For instance, a crawling policy may not achieve the same speed as a running one. Optimizing conflicting objectives of style alignment and task performance has been explored in offline RL, either by directly seeking compromises between them ( 32 ; 33 ; 61 ) , or by shifting optimal policies from one objective to the other ( 38 ) , but always at the cost of style alignment. Consequently, ensuring robust style alignment while optimizing task performance remains an open problem. In this paper, we address these challenges through the following contributions: (1) We propose a novel general view of the stylized policy learning problem as a generalization of the goal-conditioned RL (GCRL) problem ( 43 ) . (2) We instantiate our definition within the supervised data-programming framework ( 49 ) by using labeling functions as in 62 ; 59 but on trajectory windows rather than full trajectories, capturing the multi-timescale nature of styles and mitigating high credit assignment challenges by design. The use of labeling functions also allows users to quickly program various meaningful style annotations for both training data and evaluation data, making the alignment measurement easier at inference. (3) We introduce Style-Conditioned Implicit-Q-Learning (SCIQL), a style-conditioned offline RL algorithm inspired by IQL ( 26 ) which leverages advantage signals to guide the policy towards the activation of target styles, making efficient use of style-relabeling ( 47 ) and trajectory stitching ( 5 ) to allow for robust style alignment . (4) Making use of the casting of stylized policy learning problem as a RL problem, we introduce the notion of Gated Advantage Weighted Regression (GAWR) in the stylized policy learning context by using advantage functions as gates to allow style-conditioned task performance optimization . (5) We demonstrate through a set of experiments on diverse stylized RL tasks that our method effectively outperforms previous work on both style alignment and style-conditioned task performance optimization . Code, datasets and visuals are available in: https://sciql-iclr-2026.github.io/ . 2 Related work IL and offline RL. Imitation Learning seeks to learn policies by mimicking expert demonstrations, usually stored as trajectory datasets, and can be grouped into different categories, including Behavior Cloning, classical Inverse RL (IRL), and Apprenticeship / Adversarial IRL. Behavior Cloning (BC) ( 48 ) performs supervised regression of actions given states but suffers from compounding errors and distribution shifts ( 50 ) . Classical IRL ( 40 ; 15 ; 2 ) infers a reward under which the demonstration policy is optimal to optimize it via online RL. It is robust to distribution shifts but requires environment interactions. Apprenticeship / Adversarial IRL (e.g., GAIL ( 21 ) ) learns policies directly via implicit rewards, combining IRL’s robustness with BC’s direct learning, but typically requires online interactions. On the other hand, offline RL does not assume optimal demonstrations. It uses reward signals to train policies offline and tackles distribution shifts via sequence modeling ( 7 ) , biased BC ( 39 ; 16 ) , policy conservativeness ( 27 ) , expectile regression ( 26 ) , or Q-value exponential weighting ( 17 ) . In this work, we leverage offline RL techniques to jointly optimize behavior styles and task performance from reward signals, without assuming demonstration optimality. Diverse policy learning. Capturing diverse behavior from a pre-recorded dataset has been addressed in the literature under various scopes. Several methods aim to capture

Read the complete text