Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning Jinzong Dong, Wei Huang, Jianshu Zhang, Zhuo Chen, Xinzhe Yuan, Qinying Gu, Zhaohui Jiang, Nanyang Ye Abstract Offline reinforcement learning (RL) optimizes policies from a previously collected static dataset and is an important branch of RL. A popular and promising approach is to regularize actor-critic methods with behavior cloning (BC), which yields realistic policies and mitigates bias from out-of-distribution actions, but can impose an often-overlooked performance ceiling: when dataset actions are suboptimal, indiscriminate imitation structurally prevents the actor from fully exploiting high-value regions suggested by the critic, especially in later training when imitation is already dominant. We formally analyzed this limitation by investigating convergence properties of BC-regularized actor-critic optimization and verified it on a controlled continuous bandit task. To break this ceiling, we propose proximal action replacement (PAR), a plug-and-play training sample replacer that progressively replaces low-value actions with high-value actions generated by a stable actor, broadening the action exploration space while reducing the impact of low-value data. PAR is compatible with multiple BC regularization paradigms. Extensive experiments across offline RL benchmarks show that PAR consistently improves performance and approaches state-of-the-art when combined with the basic TD3+BC. Code is available at: https://github.com/NeuroDong/OfflineRL-PAR . † † affiliationnotice: Equal contribution 1 Introduction Figure 1: Effect comparison of PAR. Left: Normalized score comparison during the training process between TD3+BC and TD3+BC+PAR on Walker2d-Medium-Replay. Right: Average normalized score improvement of PAR across classic and advanced behavior cloning actor-critic methods in MuJoCo. Reinforcement learning (RL) empowers agents to develop optimal decision-making strategies through real-time interactions and has achieved remarkable progress across diverse domains ( Haarnoja et al. 2024 ) . However, these interactions with real-world environments limit their practical application to critical scenarios, such as healthcare decision-making ( Fatemi et al. 2022 ) and autonomous driving ( Diehl et al. 2023 ) . To address this challenge, offline reinforcement learning, which learns from past experiences without online interactions, has emerged as a vital and rapidly growing area of research ( Tarasov et al. 2023 ) . However, relying only on previously collected data makes offline RL suffer from the well-known distribution shift problem ( Fujimoto et al. 2019 ) , i.e., learned strategies fail to generalize to online interactions. A widely adopted family of solutions is behavior cloning (BC) regularization actor-critic, where the actor is explicitly constrained to stay close to dataset actions. Typically, it employs mean squared error (MSE) ( Fujimoto & Gu 2021 ) , Kullback-Leibler (KL) divergence ( Wu et al. 2019 ) , or maximum likelihood estimation (MLE) ( Kostrikov et al. 2022 ) to reduce the discrepancy between learned and dataset actions. Owing to its simplicity and strong performance, this behavior cloning regularization has been widely adopted, spanning offline RL approaches based on state-conditional generative modeling ( Wang et al. 2023 ; Kang et al. 2023 ) as well as some emerging offline reinforcement learning methods ( Luo et al. 2025 ; Kim et al. 2025 ) . Despite its effectiveness for stability, BC regularization can impose an often-overlooked performance ceiling when the dataset contains suboptimal actions. Intuitively, once the policy has learned to generate “realistic” actions, continued pressure to imitate suboptimal samples keeps pulling the actor toward low-value regions, even when the critic has already identified better actions. This creates a structural trade-off between imitation and optimality, which becomes increasingly limiting in later training when the main bottleneck is no longer realism but exploiting high-value actions. To formally characterize this limitation, we theoretically proved that BC regularization inherently leads to suboptimal policies by analyzing the convergence properties of alternating optimization. We further empirically validated this phenomenon through controlled experiments on a synthetic continuous bandit task, demonstrating that BC regularization structurally confines the actor to a suboptimal manifold and prevents full exploitation of high-value regions identified by the critic. To address this fundamental issue, we propose proximal action replacement (PAR), a plug-and-play training sample replacer that progressively replaces low-value dataset actions with high-value actions generated by a stable actor. By dynamically expanding the action exploration space while maintaining training stability through proximal constraints, PAR enables policies to transcend the performance ceiling imposed by BC regularization without sacrificing its stability benefits. PAR is compatible with various BC regularization forms (MSE, KL, MLE) and seamlessly integrates into existing offline RL algorithms. Extensive experiments demonstrate that PAR consistently improves performance across multiple algorithms and diverse domains, as illustrated in Fig. 1 . Our contributions are threefold: (1) Theoretical analysis: we formally prove that BC regularization structurally prevents policies from achieving optimality when dataset actions are suboptimal; (2) Methodological innovation: we propose PAR, a simple yet effective approach that addresses the suboptimality issue at the data level by progressively replacing low-value actions with high-value generated actions; and (3) Empirical validation: we demonstrate PAR’s effectiveness through comprehensive experiments across multiple offline RL algorithms and diverse domains, showing consistent performance improvements and competitive results with state-of-the-art methods. 2 Background and Related Work Offline RL : The RL problem is typically defined by a Markov decision process (MDP): M=\{S,A,P,R,\gamma,d_{0}\} , with state space S , action space A , environment dynamics P:S\times A\to\Delta(S) , reward function R:S\times A\to\mathbb{R} , discount factor \gamma , and initial state distribution d_{0} , where \Delta(S) represents the set of all probability distributions on the state space S . The goal of RL is to learn policy \pi_{\theta}(a|s) , parameterized by \theta , that maximizes the cumulative discounted reward \mathbb{E}_{s_{0}\sim d_{0},a_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim P(\cdot|s_{t},a_{t})}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\right] . The action-value (Q-value) of a policy \pi is defined as Q^{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\mid s_{0}=s,a_{0}=a\right] . In the offline RL setting, instead of the environment, a static dataset \mathcal{D}\triangleq\{(s_{t},a_{t},r_{t},s_{t+1})\} is provided. Offline RL algorithms learn a policy solely from the fixed offline dataset \mathcal{D} , without any online interaction with the environment. Behavior Cloning Regularization : Behavior cloning regularization helps models rapidly acquire realistic policies while mitigating bias from out-of-distribution actions, by constraining them to closely align with the behavior policies present in the dataset. Common behavior cloning regularizations include three forms: mean squared error (MSE) ( Fujimoto & Gu 2021 ) , Kullback–Leibler (KL) divergence ( Wu et al. 2019 ) , or maximum likelihood estimation (MLE) ( Kostrikov et al. 2022 ) , which are defined as follows. Definition 2.1 . (MSE Behavior Cloning) : For a deterministic policy \pi_{\theta}:S\rightarrow A , the MSE regularization minimizes the squared Euclidean distance between the policy’s action and the action in the dataset \mathcal{D} : \mathcal{L}_{\text{MSE}}(\theta)=\mathbb{E}_{(s,a)\sim\mathcal{D}}[\|\pi_{\theta}(s)-a\|_{2}^{2}] . Definition 2.2 . (KL Behavior Cloning) : For a stochastic policy \pi_{\theta}(\cdot|s) , the KL regularization minimizes the Kullback–Leibler divergence between \pi_{\theta} and the behavior policy \pi_{\beta} estimated from the dataset: \mathcal{L}_{\text{KL}}(\theta)=\mathbb{E}_{s\sim\mathcal{D}}[D_{\text{KL}}(\pi_{\theta}(\cdot|s)\|\pi_{\beta}(\cdot|s))] . Definition 2.3 . (MLE Behavior Cloning) : For a stochastic policy \pi_{\theta}(\cdot|s) , the MLE regularization maximizes the weighted log-likelihood of the actions in the dataset: \mathcal{L}_{\text{MLE}}(\theta)=\mathbb{E}_{(s,a)\sim\mathcal{D}}[-w(s,a)\log\pi_{\theta}(a|s)] , where w(s,a) is a weighting term often derived from Q-values or advantages. Related Work: Existing offline RL methods can be broadly categorized as follows. Behavior cloning regularization methods constrain the learned policy to stay close to the behavior policy to prevent distributional shift. TD3+BC ( Fujimoto & Gu 2021 ) adopts a minimalist approach that adds MSE behavior cloning regularization to policy updates. BRAC ( Wu et al. 2019 ) introduces a general framework that applies KL divergence regularization between the learned and behavior policies. SSAR ( Luo et al. 2025 ) proposes selective state-adaptive regularization that adjusts regularization strength based on data quality, trusting Bellman updates in high-quality states while applying constraints selectively. IQL ( Kostrikov et al. 2022 ) estimates the value of optimal actions implicitly through expectile regression and extracts the policy via advantage-weighted behavior cloning, avoiding explicit evaluation of out-of-distribution actions. Conservative value methods address extrapolation errors by penalizing out-of-distribution actions in the value function. CQL ( Kumar et al. 2020 ) learns a conservative Q-function that lower-bounds the true value function, while COMBO ( Yu et al. 2021 ) extends this conservative principle to model-based offline RL. Generative model-based methods leverage expressive policy classes to capture complex action distributions. BCQ ( Fujimoto et al. 2019 ) employs a VAE-based generative model to restrict the action space to the support of the behavior policy. Diffusion Policies ( Wang et al. 2023 ) and EDP ( Kang et al. 2023 ) represent policies as diffusion models to capture multimodal action distributions. FQL ( Park et al. 2025 ) trains an expressive flow-matching policy by distilling iterative flow processes into one-step generation, while QIPO ( Zhang et al. 2025 ) introduces energy-weighted flow matching that directly learns energy-guided flows without auxiliary models. Trajectory planning methods generate complete trajectories for long-horizon decision-making. PG ( Ki et al. 2025 ) proposes prior-guided diffusion planning that replaces the standard Gaussian prior with a learnable distribution. TAT ( Feng et al. 2024 ) aggregates information from historical and current trajectories in a dynamic tree structure to resist stochastic risks in diffusion planners. LoMAP ( Lee & Choi 2025 ) projects guided samples onto a low-rank subspace to prevent infeasible trajectory generation in diffusion planners. Conditional sequence modeling methods learn action distributions based on history trajectories and target returns. DT ( Chen et al. 2021 ) formulates reinforcement learning as a sequence modeling problem, where actions are generated conditioned on desired returns and past states. QCS ( Kim et al. 2024 ) proposes adaptive Q-aid for conditional supervised learning in offline RL, leveraging Q-values to guide sequence modeling. QT ( Hu et al. 2024 ) combines the trajectory modeling ability of Transformers with the predictability of optimal future returns from dynamic programming methods by integrating Q-value maximization into the conditional sequence modeling training loss. In contrast to these approaches, PAR takes a fundamentally different perspective: instead of modifying policy constraints, value functions, or policy representations, PAR addresses the suboptimality issue at the data level by progressively replacing low-value actions in the training batch with high-value actions generated by a stable actor. This training sample replacement strategy enables the policy to transcend the performance ceiling imposed by BC regularization while maintaining training stability through proximal constraints, without requiring modifications to the underlying algorithm architecture or policy representation. 3 Method In this section, we systematically analyze the limitations of behavior cloning regularization and introduce our proposed method, Proximal Action Replacement (PAR), to address them. First, we provide theoretical insights and toy experiments in Section 3.1 demonstrating that standard BC regularization inherently forces a trade-off between imitation and optimality, preventing the policy from fully exploiting high-value regions. Motivated by this observation, we then present PAR, a plug-and-play module that progressively replaces suboptimal samples in the training batch with high-quality generated actions, thereby expanding the exploration space while preserving training stability. 3.1 BC Regularization Leads to Suboptimal Policy Figure 2: Offline RL experiments on a simple bandit task. The backbone algorithm is TD3+BC, where BC is set to three forms: MSE, KL, and MLE. The first row shows that behavior cloning loss prevents the learned policy from moving closer to the best policy. The second row shows that PAR can significantly alleviate this situation, making the learned policy closer to the optimal policy. To understand the fundamental impact of BC regularization, we analyze the optimization dynamics from the perspective of distinguishing between the computed optimal solution (obtained by optimizing the regularized objective that balances Q-value maximization with behavior cloning constraints) and the ideal optimal solution (the true optimal policy that maximizes expected return). The formal description is presented in Theorem 3.1 . Theorem 3.1 . (Sub-optimality of BC Regularization) Let (\hat{\pi},\hat{Q}) be the computed optimal pair for the regularized objective J_{\theta}(\pi_{\theta},Q)=\mathbb{E}[\lambda Q(s,\pi_{\theta})-\mathcal{L}_{\text{MSE}}(\pi_{\theta},a_{\text{data}})] and the Bellman error. Let (\pi^{*},Q^{*}) denote the ideal optimal pair where \pi^{*}(s)=\arg\max_{a}Q^{*}(s,a) . If the dataset action a_{\text{data}} is sub-optimal (i.e., a_{\text{data}}\neq\pi^{*}(s) ) and \lambda<\infty , then the converged policy \hat{\pi} is strictly sub-optimal: (\hat{\pi},\hat{Q})\neq(\pi^{*},Q^{*})\quad\text{and}\quad Q^{*}(s,\hat{\pi}(s))\mathcal{L}_{\text{critic}} and t>T_{\text{start}} then 6: Enqueue \{(s,\pi_{\theta^{\text{sma}}}(s))\mid s\in\mathcal{B}\} into \mathcal{D}_{\text{syn}} (FIFO). 7: p\leftarrow p_{\min}+\frac{t-T_{\text{start}}}{T-T_{\text{start}}}(p_{\max}-p_{\min}) . 8: n_{\text{syn}}\leftarrow\lfloor p|\mathcal{B}|\rfloor,n_{\text{real}}\leftarrow|\mathcal{B}|-n_{\text{syn}} . 9: \mathcal{B}_{\text{sorted}}=\operatorname{sort}_{Q}(\mathcal{B}),\quad Q(b_{1})\geq\cdots\geq Q(b_{|\mathcal{B}|}) 10: Extract top- n_{\text{real}} samples to \mathcal{B}_{\text{real}} from \mathcal{B}_{\text{sorted}} . 11: Sample n_{\text{syn}} samples to \mathcal{B}_{\text{syn}} from \mathcal{D}_{\text{syn}} . 12: \mathcal{B}\leftarrow\mathcal{B}_{\text{real}}\cup\mathcal{B}_{\text{syn}} . 13: end if 14: Update \pi_{\theta} using \mathcal{B} . Policy Training 15: \theta^{\text{sma}}\leftarrow\alpha\theta^{\text{sma}}+(1-\alpha)\theta . 16: \phi^{\text{sma}}\leftarrow\alpha\phi^{\text{sma}}+(1-\alpha)\phi . 17: \mathcal{L}^{\text{sma}}\leftarrow\alpha\mathcal{L}^{\text{sma}}+(1-\alpha)\mathcal{L}_{\text{critic}} . 18: end for Fig. 2 shows the toy bandit results with PAR. The second row illustrates that PAR mitigates the anchoring effect observed in standard BC regularization: instead of converging toward the suboptimal dataset center, the learned policy shifts closer to the true optimal action. This supports the intuition that progressively replacing low-value action targets can help the actor move toward higher-value regions identified by the critic. Table 1: Comparison of normalized scores on D4RL benchmark across MuJoCo, Kitchen, and AntMaze domains. Bold values indicate that performance was improved after using PAR. Dataset BCQ BCQ +PAR TD3+BC TD3+BC +PAR IQL IQL +PAR EDP EDP +PAR SSAR SSAR +PAR HalfCheetah-M-E - - 90.7 95.2 ±0.9 86.7 93.2 ±1.1 95.8 ±0.1 93.3 ±0.5 94.9 ±1.2 96.0 ±0.3 Hopper-M-E - - 98.0 111.8 ±1.2 91.5 108.3 ±6.8 110.8 ±0.4 110.9 ±1.2 103.8 ±6.7 112.7 ±0.2 Walker2d-M-E - - 110.1 111.5 ±0.5 109.6 111.2 ±0.4 110.4 ±0.0 111.9 ±0.4 112.5 ±1.4 111.7 ±0.4 HalfCheetah-M - - 48.3 49.4 ±0.1 47.4 48.4 ±0.2 50.8 ±0.0 54.9 ±0.5 56.5 ±3.7 58.5 ±1.7 Hopper-M - - 59.3 66.7 ±2.4 66.3 71.3 ±4.3 72.6 ±0.2 86.4 ±7.6 101.6 ±0.4 102.5 ±0.4 Walker2d-M - - 83.7 85.8 ±0.6 78.3 82.5 ±4.4 86.5 ±0.2 89.5 ±0.5 87.9 ±2.4 89.3 ±0.7 HalfCheetah-M-R - - 44.6 45.5 ±0.4 44.2 44.5 ±0.3 44.9 ±0.4 48.3 ±1.3 49.6 ±0.3 49.7 ±0.3 Hopper-M-R - - 60.9 95.8 ±7.5 94.7 95.7 ±3.3 83.0 ±1.7 101.7 ±1.5 101.6 ±0.7 102.6 ±0.4 Walker2d-M-R - - 81.8 87.3 ±2.4 73.9 82.8 ±3.1 87.0 ±2.6 94.9 ±2.1 93.5 ±2.0 95.1 ±2.5 Ave. (MuJoCo) - - 75.3 83.2 ( +7.9 ) 77.0 82.0 ( +5.0 ) 82.4 88.6 ( +6.2 ) 89.1 90.9 ( +1.8 ) AntMaze-M-P 0.0 72.5 ±7.7 0.0 2.5 ±2.0 71.2 88.7 ±2.3 73.3 ±6.2 72.5 ±10.6 - - AntMaze-L-P 6.7 28.6 ±7.1 0.0 1.0 ±0.7 39.6 53.0 ±5.6 33.3 ±1.9 51.0 ±9.5 - - AntMaze-M-D 0.7 48.6 ±9.2 0.8 3.6 ±0.8 70.0 89.7 ±5.0 52.7 ±1.9 76.7 ±3.0 - - AntMaze-L-D 2.2 31.5 ±12.0 0.0 0.6 ±0.5 47.5 51.7 ±9.0 41.3 ±3.4 50.1 ±8.5 - - Ave. (AntMaze) 2.4 45.3 ( +42.9 ) 0.2 1.9 ( +1.7 ) 57.1 70.8 ( +13.7 ) 50.2 62.6 ( +12.4 ) - - Kitchen-M 8.1 25.0 ±5.4 0.0 19.6 ±3.7 51.0 57.3 ±4.7 50.2 ±1.8 64.2 ±4.3 - - Kitchen-P 18.9 24.0 ±5.4 0.0 20.4 ±4.8 46.3 65.6 ±2.8 40.8 ±1.5 60.9 ±9.5 - - Ave. (Kitchen) 13.5 24.5 ( +11.0 ) 0.0 20.0 ( +20.0 ) 48.7 61.4 ( +12.7 ) 45.5 62.5 ( +17.0 ) - - Fig. 3 further demonstrates PAR’s effectiveness in maintaining training stability. In contrast to Non-Proximal, which suffers from high-variance loss spikes after the initial decrease, PAR exhibits a smooth and stable decrease in critic loss throughout the training process. This stable behavior is achieved through PAR’s proximal constraint mechanism: by only activating action replacement when the critic loss falls below a dynamic threshold (indicating reliable value estimation), PAR ensures that the policy updates remain within a safe region where the policy divergence is bounded. The progressive replacement strategy, combined with the stability check, allows PAR to gradually expand the action exploration space while maintaining the critic’s training stability, thereby successfully overcoming the instability issue that plagues direct action replacement methods. 4 Results We evaluate PAR to answer: (1) Does it improve the behavior cloning offline reinforcement learning method? (2) Can the improved method achieve state-of-the-art results? (3) How do hyperparameters affect performance? Table 2: Comparison of normalized scores on D4RL benchmark on MuJoCo. Bold values indicate the best performance in each row, and underlined values indicate the second best. Dataset PG TAT LoMAP RefPlan FQL QIPO TD3+BC +PAR IQL +PAR EDP +PAR SSAR +PAR HalfCheetah-M-E 95.2 92.5 91.1 92.8 90.1 94.5 95.2 ±0.9 93.2 ±1.1 93.3 ±0.5 96.0 ±0.3 Hopper-M-E 110.4 109.4 110.6 57.8 86.2 108.0 111.8 ±1.2 108.3 ±6.8 110.9 ±1.2 112.7 ±0.2 Walker2d-M-E 109.4 108.8 109.2 114.0 100.5 110.9 111.5 ±0.5 111.2 ±0.4 111.9 ±0.4 111.7 ±0.4 HalfCheetah-M 45.6 44.3 45.4 74.6 60.1 54.2 49.4 ±0.1 48.4 ±0.2 54.9 ±0.5 58.5 ±1.7 Hopper-M 97.5 82.6 93.7 32.8 74.5 94.1 66.7 ±2.4 71.3 ±4.3 86.4 ±7.6 102.5 ±0.4 Walker2d-M 82.3 81.0 79.9 91.6 72.7 97.3 85.8 ±0.6 82.5 ±4.4 89.5 ±0.5 89.3 ±0.7 HalfCheetah-M-R 46.4 39.2 39.1 76.3 51.1 48.0 45.5 ±0.4 44.5 ±0.3 48.3 ±1.3 49.7 ±0.3 Hopper-M-R 91.3 95.3 97.6 82.6 85.4 101.3 95.8 ±7.5 95.7 ±3.3 101.7 ±1.5 102.6 ±0.4 Walker2d-M-R 83.7 78.2 78.7 91.2 82.1 78.6 87.3 ±2.4 82.8 ±3.1 94.9 ±2.1 95.1 ±2.5 Average (MuJoCo) 81.1 78.9 82.9 78.8 78.0 87.9 83.2 82.0 88.6 90.9 4.1 Experimental Setting Benchmarks: We evaluated PAR on D4RL benchmark tasks ( Fu et al. 2021 ) , a widely adopted standard for offline RL. Our evaluation spans various domains, including locomotion, navigation and manipulation tasks to demonstrate the method’s generalizability. Specifically, we utilize: (1) Gym-MuJoCo tasks (v2) (HalfCheetah, Hopper, Walker2d), covering three dataset qualities: Medium, Medium-Replay, and Medium-Expert; (2) AntMaze tasks (v0), which require stitching suboptimal trajectories for sparse-reward navigation, evaluating on Medium and Large maps with Play and Diverse datasets; and (3) FrankaKitchen tasks (v0), a challenging high-dimensional manipulation domain, where we use the Mixed and Partial datasets. Baselines: We compared it with some popular behavior cloning actor-critic methods, as follows: (1) BCQ ( Fujimoto et al. 2019 ) , which restricts the action space to the support of the behavior policy to minimize extrapolation error; (2) TD3+BC ( Fujimoto & Gu 2021 ) , a minimalist approach that adds a behavior cloning regularization term to the policy update; (3) IQL ( Kostrikov et al. 2022 ) , which performs implicit policy improvement via expectile regression to avoid querying out-of-distribution actions; (4) EDP ( Kang et al. 2023 ) , a diffusion-based policy method that penalizes excess diffusion variance; and (5) SSAR ( Luo et al. 2025 ) , which adopts a selective state-adaptive regularization strategy to balance policy constraint and improvement. To further demonstrate PAR’s effectiveness, we compare it with other state-of-the-art offline RL methods: (1) PG ( Ki et al. 2025 ) , which replaces the standard Gaussian prior of a behavior-cloned diffusion model with a learnable distribution optimized via behavior-regularized objectives to directly generate high-value trajectories; (2) TAT ( Feng et al. 2024 ) , which aggregates information from both historical and current trajectories in a dynamic tree-like structure to marginalize unreliable states and prioritize impactful nodes for decision-making; (3) LoMAP ( Lee & Choi 2025 ) , a training-free method that projects guided samples onto a low-rank subspace approximated from offline datasets to prevent infeasible trajectory generation; (4) RefPlan ( Lee & Choi 2025 ) , which employs local manifold approximation and projection for manifold-aware diffusion planning; (5) FQL ( Park et al. 2025 ) , which leverages an expressive flow-matching policy to model complex action distributions by training a one-step policy with RL to avoid unstable recursive backpropagation; and (6) QIPO ( Zhang et al. 2025 ) , which introduces energy-weighted flow matching that directly learns energy-guided flows without auxiliary models for offline RL. Implementation Details: We implement PAR and all baselines using PyTorch. For all baseline algorithms (BCQ, TD3+BC, IQL, EDP, and SSAR), we adopt the standard hyperparameters reported in their original papers. For PAR, we set the synthetic buffer capacity to 2000 . For all experiments, we use a batch size of 256 and report results averaged over 5 random seeds. All models are trained on NVIDIA Tesla A800 GPUs. 4.2 Experimental Results Locomotion Domain (MuJoCo): As shown in Table 1 , PAR significantly enhances all baseline methods on MuJoCo tasks. TD3+BC improves from 75.3 to 83.2 (+7.9 points), IQL from 77.0 to 82.0 (+5.0 points), EDP from 82.4 to 88.6 (+6.2 points), and SSAR from 89.1 to 90.9 (+1.8 points) on average. Notably, PAR achieves remarkable improvements on challenging Medium-Replay datasets, with Hopper-M-R showing the most dramatic gain from 60.9 to 95.8 (+34.9 points) for TD3+BC. Across all three datasets (Medium, Medium-Replay, and Medium-Expert), PAR consistently delivers performance gains, demonstrating its robustness across varying data distributions. AntMaze Domain: PAR demonstrates exceptional effectiveness on AntMaze navigation tasks, which require stitching suboptimal trajectories. As reported in Table 1 , BCQ+PAR achieves a substantial improvement from 2.4 to 45.3 (+42.9 points) on average, with particularly strong gains on AntMaze-M-P (0.0 to 72.5) and AntMaze-M-D (0.7 to 48.6). BCQ’s low baseline (avg. 2.4) stems from conservative motion limits that hinder trajectory splicing. PAR addresses this by progressively replacing low-value actions with high-value ones, expanding action exploration and enabling BCQ to overcome these restrictions. TD3+BC also exhibits a very low baseline (avg. 0.2), likely because its deterministic architecture is ill-suited for long-horizon sparse-reward tasks. While PAR can replace actions, its effectiveness is constrained by TD3+BC’s critic learning failures and limited exploration, resulting in modest gains (+1.7). IQL+PAR also shows significant improvements, advancing from 57.1 to 70.8 (+13.7 points) on average, with AntMaze-M-P improving from 71.2 to 88.7. EDP+PAR achieves competitive results, with AntMaze-M-D improving from 52.7 to 76.7. These results highlight PAR’s ability to effectively expand the exploration space in sparse-reward navigation tasks, enabling policies to discover better trajectories beyond the dataset distribution. Kitchen Domain: On the challenging FrankaKitchen manipulation tasks, PAR consistently improves all baseline methods. As shown in Table 1 , BCQ+PAR improves from 13.5 to 24.5 (+11.0 points) on average, while TD3+BC+PAR achieves substantial gains from 0.0 to 20.0, demonstrating PAR’s effectiveness even when starting from very low baseline performance. IQL+PAR advances from 48.7 to 61.4 (+12.7 points), and EDP+PAR improves from 45.5 to 62.5 (+17.0 points). Notably, on Kitchen-P, IQL+PAR achieves 65.6, representing a +19.3 point improvement over the baseline. These results validate PAR’s effectiveness in high-dimensional manipulation domains, where the method successfully guides policies toward better action sequences. Comparison with Other SOTA Methods: Table 2 compares PAR-enhanced methods against state-of-the-art offline RL approaches, including trajectory planning methods (PG, TAT, LoMAP, RefPlan) and flow-matching methods (FQL, QIPO). On MuJoCo tasks, SSAR+PAR achieves the best average performance of 90.9, followed by QIPO (87.9) and EDP+PAR (88.6). Notably, PAR-enhanced methods achieve top performance on several individual tasks: SSAR+PAR ranks first on HalfCheetah-M-E (96.0) and Hopper-M-E (112.7), while EDP+PAR and SSAR+PAR achieve the best results on Hopper-M-R (102.6) and Walker2d-M-R (95.1), respectively. These results demonstrate that PAR, when combined with strong baseline methods, can achieve competitive or superior performance compared to sophisticated SOTA approaches, while maintaining the simplicity and compatibility of the underlying algorithms. 4.3 Ablation Study To gain deeper insights into the effectiveness of PAR and understand the contribution of each component, we conducted a comprehensive ablation study. We systematically investigate two critical aspects: (1) the necessity of introducing new actions through the replacement mechanism; and (2) the hyperparameter settings. Table 3: Ablation study on the necessity of introducing new actions. Results are normalized scores averaged over 5 seeds. Bold represents the best result. Dataset Baseline Filtering PAR HalfCheetah-M-E 90.7 90.9 ±1.0 95.2 ±0.9 Hopper-M-E 98.0 102.4 ±5.4 111.8 ±1.2 Walker2d-M-E 110.1 111.2 ±0.1 111.5 ±0.5 HalfCheetah-M 48.3 48.2 ±0.2 49.4 ±0.1 Hopper-M 59.3 65.3 ±1.1 66.7 ±2.4 Walker2d-M 83.7 84.7 ±0.5 85.8 ±0.6 HalfCheetah-M-R 44.6 44.5 ±0.3 45.5 ±0.4 Hopper-M-R 60.9 76.5 ±13.7 95.8 ±7.5 Walker2d-M-R 81.8 85.2 ±1.5 87.3 ±2.4 Average 75.3 78.7 ( +3.4 ) 83.2 ( +7.9 ) Necessity of New Actions : To validate the necessity of introducing new actions through the replacement mechanism, we compare PAR with its variants without new actions. Specifically, we consider: (1) Baseline , the original TD3+BC without any modification; (2) Filtering , which retains only high-value actions from the dataset without replacement; (3) PAR , our full method that replaces low-value actions with high-value generated actions. Table 3 shows that while Filtering improves over Baseline (78.7 vs. 75.3 average), PAR significantly outperforms both methods (83.2 average), demonstrating a substantial +4.5 point gain over Filtering. This gap highlights that simply selecting high-value actions from the dataset is insufficient; introducing new high-value actions generated by the learned policy is essential for breaking the performance ceiling. Notably, on challenging tasks like Hopper-M-R, PAR achieves 95.8 compared to Filtering’s 76.5, underscoring the critical role of action replacement in expanding the exploration space beyond the dataset distribution. Hyperparameters: To determine appropriate hyperparameter settings for PAR, we conducted a grid search over the hyperparameter space. Specifically, we systematically search over three critical hyperparameters: the warm-up period T_{\text{start}} (in epochs) with values \{0,500\} , the minimum and maximum replacement ratios [p_{\min},p_{\max}] with combinations \{[0,0.3],[0,0.5]\} , and the stability threshold \beta with values \{1.1,1.5\} . Based on our experimental findings, we provide recommended hyperparameter settings in Appendix 4 to facilitate the application of PAR across different domains and tasks. 5 Discussion and Conclusion Computational Complexity Analysis: As can be seen from Algorithm 1 , the computational overhead introduced by PAR is almost negligible. Our method requires no additional model inference or training beyond the standard actor-critic updates. The only additional operations are: (1) maintaining a small FIFO queue \mathcal{D}_{\text{syn}} with a capacity of 2000 for storing synthetic state-action pairs; (2) performing simple scalar computations for computing the replacement ratio p and updating moving averages; and (3) sorting a single mini-batch of data by Q-values to identify high-value samples. The time complexity of these operations—queue operations ( O(1) amortized), scalar arithmetic ( O(1) ), and batch sorting ( O(|\mathcal{B}|\log|\mathcal{B}|) where |\mathcal{B}| is the batch size)—is negligible on modern computing hardware compared to the neural network forward and backward passes. Therefore, PAR is computationally efficient and adds minimal overhead to the base offline RL algorithms. Potential Impact, Limitations, and Future Work: PAR provides a solution as a starting point for breaking free from the constraints of offline behavior data. This approach may inspire self-training paradigms, helping offline RL algorithms surpass human-level decision-making rather than merely imitating data. However, constrained by alternating optimization and Bellman stability requirements, we are limited to self-optimization near behavior data. Therefore, future work could further explore how to stably and effectively extend the exploration space to broader and more optimal manifolds, potentially through more sophisticated stability mechanisms or alternative optimization paradigms that allow for safer exploration beyond the data distribution. Conclusion: We have identified and theoretically analyzed a fundamental limitation of BC regularization in offline RL: it structurally prevents policies from fully exploiting high-value regions when dataset actions are suboptimal. To address this, we proposed Proximal Action Replacement (PAR), a plug-and-play method that progressively replaces low-value dataset actions with high-value generated actions while maintaining training stability through proximal constraints. Extensive experiments show PAR consistently improves performance across multiple algorithms, achieving competitive or superior results while maintaining efficiency. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Proof of Theorem 3.1 Proof. We prove by contradiction. Since the optimization algorithm converges to a stationary point where the gradient vanishes, we examine whether this stationary point coincides with the optimal solution. Assume the stationary point is the optimal pair, i.e., (\hat{\pi},\hat{Q})=(\pi^{*},Q^{*}) . For this to hold, the gradient of the actor objective must vanish at \pi^{*} : \nabla_{\theta}J_{\theta}(\pi_{\theta},Q^{*})\big|_{\pi_{\theta}=\pi^{*}}=0 . Expanding the gradient: \nabla_{\theta}J_{\theta}(\pi_{\theta},Q^{*})\big|_{\pi_{\theta}=\pi^{*}}=\lambda\nabla_{\theta}Q^{*}(s,\pi_{\theta})\big|_{\pi_{\theta}=\pi^{*}}-\nabla_{\theta}\mathcal{L}_{\text{MSE}}(\pi_{\theta},a_{\text{data}})\big|_{\pi_{\theta}=\pi^{*}}. (3) Since \pi^{*} is the unconstrained maximizer of Q^{*} , we have \nabla_{\theta}Q^{*}(s,\pi^{*})=0 . Thus, the condition reduces to \nabla_{\theta}\mathcal{L}_{\text{MSE}}(\pi^{*},a_{\text{data}})=0 . However, under standard distance metrics, \nabla_{\theta}\mathcal{L}_{\text{MSE}}=0 implies \pi^{*}(s)=a_{\text{data}} , which contradicts the premise that a_{\text{data}} is sub-optimal. Therefore, the gradient at \pi^{*} is non-zero, driving the policy update away from \pi^{*} . Consequently, the stationary policy \hat{\pi} must differ from \pi^{*} . By the uniqueness of the maximum of Q^{*} , this implies Q^{*}(s,\hat{\pi}(s))0 for some a\neq\arg\max_{a^{\prime}}Q^{*}(s,a^{\prime}) ), then the converged policy \hat{\pi} is strictly sub-optimal: \hat{\pi}\neq\pi^{*}\quad\text{and}\quad\mathbb{E}_{a\sim\hat{\pi}(\cdot|s)}[Q^{*}(s,a)]<\max_{a}Q^{*}(s,a). (4) Proof. Consider the policy optimization objective with KL regularization subject to a behavior policy \pi_{\beta} : J_{\text{KL}}(\pi)=\mathbb{E}_{s\sim\mathcal{D}}\left[\mathbb{E}_{a\sim\pi_{\theta}(\cdot|s)}[Q(s,a)]-D_{\text{KL}}(\pi_{\theta}(\cdot|s)\|\pi_{\beta}(\cdot|s))\right]. (5) The closed-form optimal policy \hat{\pi} that maximizes this objective is the Boltzmann distribution: \hat{\pi}(a|s)=\frac{\pi_{\beta}(a|s)\exp\big(Q(s,a)\big)}{Z(s)},\quad\text{where }Z(s)=\int\pi_{\beta}(a^{\prime}|s)\exp\big(Q(s,a^{\prime})\big)da^{\prime}. (6) In contrast, the unregularized optimal policy \pi^{*} is deterministic (or concentrates mass on optimal actions): \pi^{*}(a|s)=\delta\big(a-\arg\max_{a^{\prime}}Q^{*}(s,a^{\prime})\big). (7) Since \pi_{\beta} has support on suboptimal actions, there exists a_{\text{sub}}\neq\arg\max_{a}Q^{*}(s,a) such that \pi_{\beta}(a_{\text{sub}}|s)>0 . Therefore, \hat{\pi}(a_{\text{sub}}|s)=\frac{\pi_{\beta}(a_{\text{sub}}|s)\exp(Q(s,a_{\text{sub}}))}{Z(s)}>0 . Now, consider the expected Q-value under \hat{\pi} : \displaystyle\mathbb{E}_{a\sim\hat{\pi}(\cdot|s)}[Q^{*}(s,a)] \displaystyle=\int\hat{\pi}(a|s)Q^{*}(s,a)da (8) \displaystyle=\frac{1}{Z(s)}\int\pi_{\beta}(a|s)\exp(Q(s,a))Q^{*}(s,a)da (9) \displaystyle=\sum_{a}\hat{\pi}(a|s)Q^{*}(s,a). (10) Let a^{*}=\arg\max_{a}Q^{*}(s,a) denote the optimal action. Since \hat{\pi} assigns non-zero probability to suboptimal actions (specifically a_{\text{sub}} ), and Q^{*}(s,a_{\text{sub}})0 and Q^{*}(s,a_{\text{sub}})0 is a bounded weighting function (i.e., w(s,a)<\infty for all (s,a) ). Let (\pi^{*},Q^{*}) denote the ideal optimal pair where \pi^{*}(a|s)=\delta(a-\arg\max_{a^{\prime}}Q^{*}(s,a^{\prime})) . If the dataset \mathcal{D} contains suboptimal actions (i.e., there exists (s,a)\in\mathcal{D} such that a\neq\arg\max_{a^{\prime}}Q^{*}(s,a^{\prime}) ), then the converged policy \hat{\pi} is strictly sub-optimal: \hat{\pi}\neq\pi^{*}\quad\text{and}\quad\mathbb{E}_{a\sim\hat{\pi}(\cdot|s)}[Q^{*}(s,a)]<\max_{a}Q^{*}(s,a). (16) Proof. For MLE-based methods, the policy is updated by maximizing the weighted log-likelihood: \hat{\theta}=\arg\max_{\theta}\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[w(s,a)\log\pi_{\theta}(a|s)\right], (17) where w(s,a)>0 denotes the weight. The optimal solution \hat{\pi} converges to the weighted behavior distribution: \hat{\pi}(a|s)=\frac{w(s,a)\pi_{\beta}(a|s)}{\sum_{a^{\prime}}w(s,a^{\prime})\pi_{\beta}(a^{\prime}|s)}. (18) Since w(s,a)<\infty for all (s,a) and the dataset contains suboptimal actions, the support of \hat{\pi} is contained in the support of \pi_{\beta} : \text{supp}(\hat{\pi}(\cdot|s))\subseteq\text{supp}(\pi_{\beta}(\cdot|s)) . Let a^{*}=\arg\max_{a}Q^{*}(s,a) denote the optimal action. We consider two cases: Case 1: If \pi_{\beta}(a^{*}|s)=0 (optimal action not in dataset), then \hat{\pi}(a^{*}|s)=0 , and clearly \hat{\pi}\neq\pi^{*} since \pi^{*}(a^{*}|s)=1 . Moreover, since \hat{\pi} assigns zero probability to the optimal action, we have: \mathbb{E}_{a\sim\hat{\pi}(\cdot|s)}[Q^{*}(s,a)]=\sum_{a\neq a^{*}}\hat{\pi}(a|s)Q^{*}(s,a)0 but there exists a_{\text{sub}}\neq a^{*} with \pi_{\beta}(a_{\text{sub}}|s)>0 and w(s,a_{\text{sub}})>0 (which must hold since the dataset contains suboptimal actions), then: \hat{\pi}(a_{\text{sub}}|s)=\frac{w(s,a_{\text{sub}})\pi_{\beta}(a_{\text{sub}}|s)}{\sum_{a^{\prime}}w(s,a^{\prime})\pi_{\beta}(a^{\prime}|s)}>0. (20) Since \hat{\pi} assigns non-zero probability to suboptimal actions, we have: \displaystyle\mathbb{E}_{a\sim\hat{\pi}(\cdot|s)}[Q^{*}(s,a)] \displaystyle=\sum_{a}\hat{\pi}(a|s)Q^{*}(s,a) (21) \displaystyle=\hat{\pi}(a^{*}|s)Q^{*}(s,a^{*})+\sum_{a\neq a^{*}}\hat{\pi}(a|s)Q^{*}(s,a) (22) \displaystyle=\hat{\pi}(a^{*}|s)Q^{*}(s,a^{*})+\hat{\pi}(a_{\text{sub}}|s)Q^{*}(s,a_{\text{sub}})+\sum_{a\neq a^{*},a\neq a_{\text{sub}}}\hat{\pi}(a|s)Q^{*}(s,a) (23) \displaystyle<\hat{\pi}(a^{*}|s)Q^{*}(s,a^{*})+\hat{\pi}(a_{\text{sub}}|s)Q^{*}(s,a^{*})+\sum_{a\neq a^{*},a\neq a_{\text{sub}}}\hat{\pi}(a|s)Q^{*}(s,a^{*}) (24) \displaystyle=Q^{*}(s,a^{*})\sum_{a}\hat{\pi}(a|s) (25) \displaystyle=\max_{a}Q^{*}(s,a), (26) where the strict inequality holds because \hat{\pi}(a_{\text{sub}}|s)>0 and Q^{*}(s,a_{\text{sub}})