IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic
Stefano Viel*, Luca Viano*, Volkan Cevher · 2025
Read IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic by Stefano Viel*, Luca Viano*, Volkan Cevher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19859.txt.
Introduction
Learning a skill from watching an expert is hard when the learner must also probe the world on its own. Stefano Viel, Luca Viano, and Volkan Cevher present SOAR in 2025, a frame for imitation learning that pairs a learner with several critics. The critics do not just score moves; their spread of views shows how unsure the system is.
That doubt is used with intent: the learner leans toward hope, which drives it to explore where scores may be better than they look. The method swaps updates of cost and policy in turn. In a simple table setting, the team proves bounds that match the best known results. In rich test worlds, the frame lifts the scores of three known methods and halves the episodes they need.
Catalog record
| CREATOR | Stefano Viel*, Luca Viano*, Volkan Cevher |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-19859
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic Stefano Viel*, Luca Viano*, Volkan Cevher Abstract This paper introduces the SOAR framework for imitation learning. SOAR is an algorithmic template that learns a policy from expert demonstrations with a primal dual style algorithm that alternates cost and policy updates. Within the policy updates, the SOAR framework uses an actor critic method with multiple critics to estimate the critic uncertainty and build an optimistic critic fundamental to drive exploration. When instantiated in the tabular setting, we get a provable algorithm with guarantees that matches the best known results in the desired accuracy parameter ϵ \epsilon . Practically, the SOAR template can boost the performance of any imitation learning algorithm based on Soft Actror Critic (SAC). As an example, we show that SOAR can boost consistently the performance of the following SAC-based imitation learning algorithms: f f -IRL, ML-IRL and CSIL. Overall, thanks to SOAR, the required number of episodes to achieve the same performance is reduced by half. 0 0 0 Project code available at https://github.com/stefanoviel/SOAR-IL/tree/master 1 Introduction Several recent state of the art imitation learning (IL) algorithms [ 55 , 115 , 26 , 106 , 102 ] are built on Soft Actor Critic (SAC) [ 28 ] to perform the policy updates. SAC uses entropy regularized policy updates to maintain a strictly positive probability of taking each action. However, this is known to be an inefficient exploration strategy if deployed alone [ 10 ] . Indeed, several recent theoretical imitation learning achieve performance guarantees by adding exploration bonuses on top of the regularized policy updates, which encourage the learner to visit state-action pairs that have not been visited previously. Unfortunately, such works are only available in the tabular setting [ 85 , 110 ] and in the linear setting [ 103 ] . The design of the exploration bonuses in these works is strictly tight to the tabular or linear structure of the transition dynamics, therefore, these analyses offer little insight on how to design an efficient exploration mechanism using neural network function approximation. There is, therefore, a lack of a technique that satisfies the following two requirements. • It is statistically and computationally efficient in the tabular setting. • It can be implemented easily in continuous states and actions problems requiring neural networks function approximation. In this paper, we present a general template, dubbed Soft Optimistic Actor cRitic Imitation Learning (SOAR-IL) satisfying these requirements. The main idea is to act according to an optimistic critic within the SAC block on which many IL algorithms rely. Here, optimism means appropriately underestimating the expected cumulative cost incurred by playing a policy in the environment. This principle known as optimism in the face of uncertainty has led to several successful algorithms in the bandits community. Figure 1 : Summary of experimental results. Each plot compares the average normalized return across 4 MuJoCo environments with 16 expert trajectories for a base algorithm and its SOAR-enhanced version. SOAR replaces the single critic in SAC-based methods with multiple critics to compute an optimistic estimate. Across all algorithms, incorporating SOAR consistently improves performance. ML-IRL (SA) stands for ML-IRL [ 115 ] from expert state-action demonstrations. While optimism is often achieved using the structure of the problem (tabular, linear, etc.), in this work, we build optimistic estimators using an ensemble technique. That is, multiple estimators for the same quantity are maintained and aggregated to obtain an optimistic estimator. This technique scales well with deep imitation learning. To summarize, we have the following contributions. Theoretical contribution We show that there exists a computationally efficient algorithm that uses an ensemble based exploration technique that gives access to \mathcal{O}(\epsilon^{-2}) expert trajectories and \mathcal{O}(\epsilon^{-2}) interactions in a tabular MDP outputs a policy such that its cumulative expected cost is at most \epsilon higher than the expert cumulative expected cost with high probability. Practical Contribution We apply an ensemble-based exploration technique, SOAR, to boost the performance of deep imitation learning algorithms built on SAC, demonstrating its effectiveness on MuJoCo environments. Specifically, we show that incorporating SOAR consistently boosts the performance of base methods such as Coherent Soft Imitation Learning (CSIL) [ 106 ] , Maximum Likelihood IRL (ML-IRL) [ 115 ] and RKL [ 55 ] . As shown in Figure 1 , our approach consistently outperforms the base algorithms across all MuJoCo environments. Notably, SOAR achieves the best performance of the baselines requiring only approximately half the number of learning episodes. 2 Preliminaries and Notation The environment is abstracted as Markov Decision Process (MDP) [ 70 ] which consists of a tuple (\mathcal{S},\mathcal{A},P,c,\boldsymbol{\nu}_{0},\gamma) where \mathcal{S} is the state space, \mathcal{A} is the action space, P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta_{\mathcal{S}} is the transition kernel, that is, P(s^{\prime}|s,a) denotes the probability of landing in state s^{\prime} after choosing action a in state s . Moreover, \boldsymbol{\nu}_{0} is a distribution over states from which the initial state is sampled. Finally, c:\mathcal{S}\times\mathcal{A}\rightarrow[0,1] is the cost function, and \gamma\in[0,1) is called the discount factor. Value functions and occupancy measures We define the state value function at state s\in\mathcal{S} for the policy \pi under the cost function c as V_{c}^{\pi}(s)\triangleq\mathbb{E}\left[{\sum^{\infty}_{h=0}\gamma^{h}c(s_{h},a_{h})|s_{1}=s}\right] . The expectation over both the randomness of the transition dynamics and the one of the learner’s policy. Another convenient quantity is the occupancy measure of a policy \pi denoted as d^{\pi}\in\Delta_{\mathcal{S}\times\mathcal{A}} and defined as follows d^{\pi}(s,a)\triangleq(1-\gamma)\sum^{\infty}_{h=0}\gamma^{h}\mathbb{P}\left[{s,a\text{ is visited after $h$ steps acting with $\pi$}}\right] . We can also define the state occupancy measure as d^{\pi}(s)\triangleq(1-\gamma)\sum^{\infty}_{h=0}\gamma^{h}\mathbb{P}\left[{s\text{ is visited after $h$ steps acting with $\pi$}}\right] . Imitation Learning In imitation learning, the learner is given a dataset \mathcal{D}_{{\pi_{\textup{E}}}} of expert trajectories collected by an unknown expert policy {\pi_{\textup{E}}} . 1 1 1 In order, to accommodate state-only and state-action with a unified analysis we overload the notation for the expert dataset. \mathcal{D}_{\pi_{\textup{E}}} denotes a collection of samples from the expert state occupancy measure in the former case and a collection of state-actions sampled from the state-action occupancy measure in the latter case. By trajectory \boldsymbol{\tau}^{k} , we mean the sequence of states and actions sampled rolling out the policy \pi_{k} for a number of steps sampled from the distribution \mathrm{Geometric}(1-\gamma) . Given \mathcal{D}_{{\pi_{\textup{E}}}} , the learner adopts an algorithm \mathcal{A} to learn a policy \pi^{\mathrm{out}} such that \left\langle{\boldsymbol{\nu}_{0}},{V^{\widehat{\pi}^{k}}_{c_{\mathrm{true}}}-V^{{\pi_{\textup{E}}}}_{c_{\mathrm{true}}}}\right\rangle\leq\epsilon with high probability. We use the notation \mathbf{e}_{s} to denote a vector in \mathbb{R}^{\left|{\mathcal{S}}\right|} zero everywhere but in the coordinate corresponding to the state s ( for an arbitrary ordering of the states). Analogously, we use \mathbf{e}_{s,a} to denote a vector in \mathbb{R}^{\left|{\mathcal{S}}\right|\left|{\mathcal{A}}\right|} zero everywhere but in the (s,a)^{th} entry which equals one. 3 The Algorithm Algorithm 1 SOAR-Imitation Learning 0: Reward step size \alpha , Expert dataset \mathcal{D}_{\pi_{\textup{E}}} , Discount factor \gamma , Policy step size \eta . 1: Initialize \pi^{1} as uniform distribution over \mathcal{A} . 2: Initialize empty replay buffer,i.e. \mathcal{D}^{0}=\left\{{}\right\} 3: for k=1 to K do 4: \tau^{k}\leftarrow\textsc{CollectTrajectory}(\pi^{k}) 5: Add \tau^{k} to replay buffer,i.e. \mathcal{D}^{k}=\mathcal{D}^{k-1}\cup\tau^{k} . 6: c^{k}\leftarrow\textsc{UpdateCost}(c^{k-1},\mathcal{D}_{\pi_{\textup{E}}},\mathcal{D}^{k},\alpha) 7: for \ell=1 to L do 8: Compute estimator Q^{k}_{\ell} . 9: end for 10: Q^{k}=\textsc{OptimisticQ}(\left\{{Q^{k}_{\ell}}\right\}^{L}_{\ell=1}) . 11: \pi^{k}(a|s)=\textsc{PolicyUpdate}(\eta,\left\{{Q^{\tau}(s,a)}\right\}^{k}_{\tau=1}) 12: end for In this Section, we describe Algorithm 1 . A meta-algorithm that encompasses several existing imitation learning algorithms. Inside each iteration of the main for loop, the learner collects a