Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows
Chenyu Yang ∗ , Denis Tarasov ∗ , Davide Liconti, Hehui Zheng, Robert K. Katzschmann · 2026
Read Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows by Chenyu Yang ∗ , Denis Tarasov ∗ , Davide Liconti, Hehui Zheng, Robert K. Katzschmann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09580.txt.
Introduction
Fine-tuning dexterous robot policies in the real world is hard: interaction budgets are small and action distributions are highly multimodal. Chenyu Yang and five co-authors present SERNF, a 2026 sample-efficient off-policy fine-tuning framework built on normalizing flows. Diffusion policies cannot support conservative likelihood-based updates since their action odds are intractable, while Gaussian policies collapse under multimodality, especially for chunked actions.
A normalizing flow policy yields exact likelihoods for multimodal action chunks, enabling stable updates through likelihood regularization. An action-chunked critic scores whole action sequences, aligning value estimates with the policy's temporal structure for better long-horizon credit assignment. Tested on cutting tape with scissors from a case and rotating a cube in hand palm-down, the framework adapts stably and efficiently where standard methods struggle.
Catalog record
| CREATOR | Chenyu Yang ∗ , Denis Tarasov ∗ , Davide Liconti, Hehui Zheng, Robert K. Katzschmann |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-09580
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows Chenyu Yang ∗ , Denis Tarasov ∗ , Davide Liconti, Hehui Zheng, Robert K. Katzschmann Abstract Real-world fine-tuning of dexterous manipulation policies remains challenging due to limited real-world interaction budgets and highly multimodal action distributions. Diffusion-based policies, while expressive, do not permit conservative likelihood-based updates during fine-tuning because action probabilities are intractable. In contrast, conventional Gaussian policies collapse under multimodality, particularly when actions are executed in chunks, and standard per-step critics fail to align with chunked execution, leading to poor credit assignment. We present SOFT-FLOW, a sample-efficient off-policy fine-tuning framework with normalizing flow (NF) to address these challenges. The normalizing flow policy yields exact likelihoods for multimodal action chunks, allowing conservative, stable policy updates through likelihood regularization and thereby improving sample efficiency. An action-chunked critic evaluates entire action sequences, aligning value estimation with the policy’s temporal structure and improving long-horizon credit assignment. To our knowledge, this is the first demonstration of a likelihood-based, multimodal generative policy combined with chunk-level value learning on real robotic hardware. We evaluate SOFT-FLOW on two challenging dexterous manipulation tasks in the real world: cutting tape with scissors retrieved from a case, and in-hand cube rotation with a palm-down grasp – both of which require precise, dexterous control over long horizons. On these tasks, SOFT-FLOW achieves stable, sample-efficient adaptation where standard methods struggle. I Introduction Robotic manipulation systems increasingly rely on high-capacity visuomotor policies trained from large offline datasets. While these policies can achieve impressive performance in controlled settings, deploying them reliably in the real world remains challenging, as manipulation policies must operate across a wide range of environments and conditions that inevitably change over time. In practice, real-world deployment is affected by numerous sources of mismatch, such as unmodeled dynamics, hardware errors, camera drift, domain shift, and task-specific issues that are difficult to capture during data collection. These challenges heavily affect dexterous and precise manipulation tasks, where collecting large amounts of high-quality demonstration data is expensive in terms of both equipment and time. Such tasks are often hard to teleoperate, require expert skill, and are sensitive to small variations in contact and timing, making scalable data collection costly and error-prone. As a result, roboticists frequently deploy policies that “almost” work, showing reasonable behavior but not achieving the reliability and success rates required for real-world, in-the-wild scenarios. In contrast to the prevailing paradigm of scaling real-robot datasets, we target settings where only limited task-specific data is available and fine-tuning must extract maximal improvement from each additional rollout. In this setting, sample efficiency is a critical requirement for policy fine-tuning algorithms. Since real-world robot interaction time is costly, an effective algorithm and architecture should operate under a tight interaction budget and make maximum use of each collected sample. Moreover, to enable RL updates, the policy should be formulated probabilistically, allowing optimization through likelihood-based objectives. We propose to represent the policy with a normalizing flow (NF) and to learn an action-chunked critic that evaluates temporally-extended action sequences (chunks). This combination aims to (i) keep the policy expressive yet tractable for likelihood-based regularization and stable updates, and (ii) improve credit assignment and variance reduction when the control interface executes action chunks. We evaluate on two real-world dexterous manipulation tasks chosen to reflect two dominant deployment pipelines in robotics. (i) Scissors retrieval and tape cutting start from a small set of human teleoperation demonstrations, representing the common imitation-learning workflow for contact-rich tasks where simulation is difficult. (ii) In-hand cube rotation starts from a policy trained in simulation and distilled into our architecture, representing the mainstream sim-to-real reinforcement-learning workflow for dexterous hands. Both tasks are long-horizon, high-dimensional, and sensitive to small contact and timing errors, making them strong stress tests for sample-efficient fine-tuning. The main contributions of this study can be summarized as follows: • We introduce an RL fine-tuning method for real-world visuomotor control that combines normalizing-flow policies with an action-chunked critic . • We describe a practical training recipe for limited on-robot data , including conservative regularization toward the initial policy. • We empirically evaluate our method on representative dexterous and precise real-world tasks and analyze the impact of chunk length and critic design. II Related Work Policy optimization for generative policies Diffusion and flow-matching policies enable expressive, multimodal action distributions for imitation learning [ 6 , 10 , 51 ] , but their use in reinforcement learning is limited by intractable action likelihoods, which hinder conservative or likelihood-regularized fine-tuning. Recent work connects diffusion- or flow-based models with policy optimization [ 2 , 23 , 34 , 41 , 52 ] . In contrast, we focus on normalizing flows , which retain comparable expressiveness while providing exact likelihoods, enabling stable and principled RL-based fine-tuning. Normalizing flows for RL and IL Normalizing flows have recently been shown to be effective policy representations for reinforcement and imitation learning [ 18 , 29 , 49 ] , but their application to real robotic systems remains limited. Prior real-world deployments are restricted to on-policy settings [ 25 ] , while most existing work focuses on simulation [ 1 , 9 , 33 ] . Our work leverages normalizing-flow policies for visuomotor control and focuses on stable, conservative fine-tuning on real hardware. We are first to apply normalizing flows within an off-policy reinforcement learning framework for real-world robotic fine-tuning. Temporal abstraction and action chunking Temporally extended action sequences are now standard in robotic control, enabling real-time execution and reduced feedback frequency [ 6 ] . Prior work shows that aligning critics with action chunking improves learning efficiency and credit assignment [ 28 ] . We extend these ideas by integrating action-chunked critics into a real-world fine-tuning pipeline. Offline + online RL Real-world robot learning commonly combines offline initialization with limited online fine-tuning. AWAC [ 37 ] and related methods enable off-policy adaptation but lack explicit conservatism in highly multimodal action spaces. Subsequent work studies the offline-to-online transition [ 3 , 38 ] , trading off conservatism and exploration at the cost of increased algorithmic complexity. Systems approaches such as IBRL and SERL emphasize the practical integration of demonstrations, off-policy RL, and tooling for real robots [ 21 , 31 ] , while extreme online-only efficiency has been demonstrated in locomotion from scratch [ 45 ] . In contrast, SOFT-FLOW targets sample-efficient real-world fine-tuning of pretrained visuomotor policies under multimodal action distributions for sparse reward tasks by combining exact-likelihood generative policies with chunk-aligned value learning. III Background MDP with observations and action chunks We consider an episodic, discounted Markov decision process (MDP) \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma,T) , with state space s\in\mathcal{S} , continuous actions a_{t}\in\mathcal{A} , transition function P:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S} , sparse reward function r , discount factor \gamma , and horizon T . The agent receives observations o_{t}\in\mathcal{O} and selects an action chunk a_{t:t+H} , which is executed open-loop over the next H environment steps. Following 28 , the H -step return is defined as \begin{aligned} V_{H\text{-step}}(s_{t})=\sum_{\tau=t}^{t+H-1}\gamma^{\tau-t}r_{\tau}+\gamma^{H}Q_{H\text{-step}}(s_{t+H},a_{t+H:t+2H}),a_{t+H:t+2H}\sim\pi(s_{t+H}),\end{aligned} (1) where Q_{H\text{-step}} is an unbiased action-value function over H -step action chunks. This formulation propagates value information back over H steps at once and better matches modern imitation and visuomotor learning frameworks in which policies generate temporally extended action sequences. Offline RL and multi-step bootstrapping In offline reinforcement learning