Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings

Seyed Mahdi Basiri Azad, Joschka Boedecker · 2025

Read Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings by Seyed Mahdi Basiri Azad, Joschka Boedecker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24432.txt.

Introduction

Seyed Mahdi Basiri Azad and Joschka Boedecker propose a simple way to speed up Q-learning when rewards are rare. In sparse reward settings, agents get little feedback and waste much time exploring blindly. Their method, Fill in the Blanks, uses a handful of successful demonstrations to initialize the agent's value function before online learning begins.

Value estimates are precomputed from the offline demonstrations and used as targets early in training, giving the agent a useful prior over promising actions. The agent then refines those estimates through normal interaction. Experiments on benchmark tasks show much lower exploration burden and better sample efficiency than learning from scratch.

Catalog record

CREATORSeyed Mahdi Basiri Azad, Joschka Boedecker
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24432

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings Seyed Mahdi Basiri Azad, Joschka Boedecker Abstract Reinforcement learning (RL) in sparse-reward environments remains a significant challenge due to the lack of informative feedback. We propose a simple yet effective method that uses a small number of successful demonstrations to initialize the value function of an RL agent. By precomputing value estimates from offline demonstrations and using them as targets for early learning, our approach provides the agent with a useful prior over promising actions. The agent then refines these estimates through standard online interaction. This hybrid offline-to-online paradigm significantly reduces the exploration burden and improves sample efficiency in sparse-reward settings. Experiments on benchmark tasks demonstrate that our method accelerates convergence and outperforms standard baselines, even with minimal or suboptimal demonstration data. 1 Introduction In many real-world applications of reinforcement learning (RL), agents must operate in environments where informative feedback is sparse or delayed. Sparse reward settings—where success is indicated by a binary signal, such as task completion or failure—pose a significant challenge for standard RL algorithms. In such environments, agents often receive no meaningful signal for extended periods, leading to slow and inefficient learning unless aided by strong inductive biases or extensive exploration. A promising strategy to address this issue is to incorporate demonstration data. Demonstrations can guide agents toward reward-yielding regions of the state space, thereby reducing the need for blind exploration. However, existing approaches frequently assume access to a large set of expert trajectories or require detailed reward annotations—assumptions that are often unrealistic in practical deployments. In this work, we introduce a simple yet effective method that jump-starts value learning using only a handful of successful demonstrations—potentially as few as one. Our approach makes no assumptions about demonstration optimality or access to dense reward signals. As long as the demonstrations achieve the task goal, they can offer valuable guidance. By estimating state value functions from demonstration data and using these estimates as targets for temporal difference (TD) updates, our method helps the agent focus its exploration early on. Once the agent begins interacting with the environment, it continues to refine its value estimates through standard TD learning in an online fashion. Our method offers several key advantages: 1. It is effective with minimal demonstration data—even a single successful trajectory can suffice. 2. It accommodates sub-optimal demonstrations, as long as they achieve the task goal. 3. It enables efficient learning in sparse-reward environments with only binary success/failure signals. We first motivate our approach through a regret analysis of \epsilon -greedy policies, highlighting how initial value estimates from demonstrations can reduce exploration cost. We then validate our method empirically across standard sparse-reward benchmarks, demonstrating substantial improvements in learning speed compared to vanilla Q-learning and common initialization baselines. Our results suggest that even minimal, imperfect demonstrations—if leveraged effectively—can dramatically improve sample efficiency in value-based RL. 2 Related Works We review relevant literature on offline RL, fine-tuning from demonstrations, sparse reward settings, and learning from sub-optimal demonstrations. 2.1 Offline RL from Demonstrations Offline RL methods [ 11 , 13 , 7 ] learn policies solely from static data, leveraging rewards to potentially surpass the demonstrator. In contrast, imitation learning [ 3 , 21 , 23 ] only mimics observed actions. However, offline RL often assumes access to a well-defined reward function and large, diverse datasets—both costly in practice. Without environment interactions, these methods are also limited in their ability to improve beyond the demonstrations and tend to act conservatively to avoid out-of-distribution actions [ 13 ] . We instead use a few successful demonstrations to initialize Q-values before transitioning to online learning. This hybrid approach reduces the need for extensive offline data and enables continual policy improvement through interaction. 2.2 Offline-to-Online RL (Warm-Starting / Fine-Tuning) Fine-tuning offline agents using online data aims to enhance robustness and generalization. However, most approaches rely on large offline datasets and suffer performance drops during the offline-to-online transition [ 12 , 10 , 16 ] , especially in sparse reward settings where mismatches in reward signals can destabilize value learning. Our method avoids these pitfalls by using only a few successful demonstrations and explicitly supporting sparse rewards. By directly assigning value estimates to demonstrated states, we ensure a stable transition to online learning without reward mismatches or performance regressions. 2.3 RL with Sparse Rewards Sparse reward settings simplify reward design but complicate learning due to infrequent feedback. Exploration methods using intrinsic rewards [ 18 , 22 , 15 , 25 ] encourage coverage but may not guide agents toward task completion. Reward relabeling methods [ 1 , 12 , 8 , 2 ] generate synthetic goals, train discriminators between expert and non-expert data, or use expert visitation counts to densify feedback, but at the cost of added complexity and computation. Instead, we reuse real demonstrations to increase informative transitions without altering the reward function or training auxiliary models. This preserves simplicity while facilitating goal-directed learning. 2.4 Learning from Sub-optimal Demonstrations Since optimal demonstrations are hard to collect, many methods relax this requirement, using successful but sub-optimal trajectories [ 20 , 4 ] . These methods often schedule a gradual shift from imitation to reinforcement learning. However, sparse rewards combined with limited demonstrations can lead to weak initial policies and hinder value propagation. Ranking-based approaches [ 6 , 14 , 19 ] attempt to prioritize better trajectories but require many demonstrations and reliable ranking signals. In contrast, our method requires only a few successful (not necessarily optimal) demonstrations. By precomputing value estimates for demonstrated states, we enable effective bootstrapping and allow online interactions to refine these initial estimates—without ranking or heavy data requirements. 3 Background We now introduce the notation and review value-based reinforcement learning concepts. 3.1 Notation We consider settings where the environment is represented by a Markov Decision Process (MDP) and is defined as a tuple \mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{T},r,\gamma,\mu_{0}) . \mathcal{S} and \mathcal{A} represent the state and action spaces respectively. \mathcal{T}(s^{\prime}|s,a) represents the state transition dynamics, r(s,a) represents the reward function, \gamma\in(0,1] is the discount factor and \mu_{0} represents the starting state distribution. As in offline reinforcement learning setting, we only have access to a limited set of demonstrations of the form \mathcal{D}=\{(s_{0},a_{0},s_{1},a_{1},...s_{T})^{i}\}_{i=0}^{N} . In this paper, we are focusing on a limited setting where neither the transition dynamics \mathcal{T}(s^{\prime}|s,a) nor the dense reward function r(s,a) are known. The environment instead provides a sparse binary reward representing the success of failure of the agent in accomplishing a task. 3.2 Reinforcement Learning In reinforcement learning (RL), an agent interacts with an environment modeled as a Markov Decision Process (MDP), aiming to learn a policy \pi(a|s) that maximizes the expected cumulative discounted reward over time. The agent selects actions a\in\mathcal{A} based on the current state s\in\mathcal{S} , receives a reward r(s,a) , and transitions to a new state s^{\prime}\sim\mathcal{T}(s^{\prime}|s,a) . The return from a state s_{t} is defined as the total discounted reward accumulated from time step t onward: G_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}, where \gamma\in(0,1] is a discount factor that prioritizes immediate rewards over distant ones. To evaluate policies and guide decision-making, RL defines two primary value functions: • The state-value function V^{\pi}(s) gives the expected return when starting in state s and following policy \pi thereafter:

Read the complete text