Quantization-Free Autoregressive Action Transformer

Ziyad Sheebaelhamd, Michael Tschannen, Michael Muehlebach, Claire Vernade · 2025

Read Quantization-Free Autoregressive Action Transformer by Ziyad Sheebaelhamd, Michael Tschannen, Michael Muehlebach, Claire Vernade (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14259.txt.

Introduction

Ziyad Sheebaelhamd, Michael Tschannen, Michael Muehlebach, and Claire Vernade published a paper in 2025 on teaching robots by imitation without chopping actions into a fixed codebook. Most transformer policies first quantize smooth movements into discrete tokens, which breaks the natural continuity of motion. Their method plugs a generative transformer that speaks in continuous values directly into the policy.

The change simplifies the training pipeline while keeping the step-by-step prediction style that makes transformers strong. On a range of standard simulated robot tasks the method matched the best published results. The authors also study how the way actions are sampled at run time changes performance, and tune it for further gains. Smooth actions, represented smoothly, seem to imitate better.

Catalog record

CREATORZiyad Sheebaelhamd, Michael Tschannen, Michael Muehlebach, Claire Vernade
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantization-Free Autoregressive Action Transformer Ziyad Sheebaelhamd, Michael Tschannen, Michael Muehlebach, Claire Vernade Abstract Current transformer-based imitation learning approaches introduce discrete action representations and train an autoregressive transformer decoder on the resulting latent code. However, the initial quantization breaks the continuous structure of the action space thereby limiting the capabilities of the generative model. We propose a quantization-free method instead that leverages Generative Infinite-Vocabulary Transformers (GIVT) as a direct, continuous policy parametrization for autoregressive transformers. This simplifies the imitation learning pipeline while achieving state-of-the-art performance on a variety of popular simulated robotics tasks. We enhance our policy roll-outs by carefully studying sampling algorithms, further improving the results. † † footnotetext: Correspondence: Ziyad Sheebaelhamd <[email protected]> 1 Introduction Generative modeling lies at the heart of modern machine learning, enabling the production of outputs that adhere to user-specified objectives and constraints by navigating high-dimensional spaces of potential outcomes. Fundamentally, this process can be understood as a form of search where naive or exhaustive strategies are computationally intractable due to the size of the space. We consider imitation learning for robotic systems as an extreme example, where the search space has uncountable cardinality and a complex structure arising from nonlinear dynamics, interactions with the environment and the fundamental multimodality in human or robot behavior. In many real-world robotics applications, the control signals involved are continuous in nature [ 33 , 59 , 28 ] . Current state-of-the-art methods for modeling policies in continuous control tasks broadly fall into two categories: autoregressive transformer models [ 26 , 45 ] and diffusion models [ 7 , 54 , 40 ] , which will be discussed in the next two paragraphs. Existing autoregressive policies, on the one hand, ignore the challenge of learning in a continuous domain by discretizing actions [ 26 , 45 ] . This discretization can introduce several drawbacks: It discards the inherent structure of the continuous space, increases complexity by adding a separate quantization step, and may limit expressiveness or accuracy when fine-grained control is required. Furthermore, the popular approach of learning a VQ-VAE [ 52 ] of the action space involves nondifferentiable operations and therefore requires advanced tricks during representation learning to sidestep optimization difficulties and instabilities [ 24 , 20 , 18 , 31 ] . Hence, our goal is to design autoregressive transformers that preserve the continuous nature of the action space, leveraging the smooth and potentially multimodal behavior patterns that experts exhibit. Diffusion policies, on the other hand, are adept at capturing continuous action distributions, but face two critical challenges in control tasks. First, real-time applications like robotics demand fast, efficient inference, yet diffusion models rely on iterative sampling, requiring dozens of forward passes to generate actions. This makes them orders-of-magnitude slower than autoregressive or feedforward policies [ 26 ] . Second, the downstream task of imitation learning is often reinforcement learning [ 43 , 39 , 58 , 17 ] , which typically requires exact action likelihoods for tasks like exploration and off-policy corrections [ 49 , 1 , 27 , 56 ] . Diffusion models, however, generate samples by integrating (stochastic) differential equations [ 48 , 47 ] , making likelihood estimation computationally expensive. To address these limitations, we propose the use of a Generative Infinite Vocabulary Transformer (GIVT) [ 50 ] that models the policy as a Gaussian Mixture Model (GMM) on top of a decoder-only transformer architecture [ 53 , 41 ] . This approach enables modeling continuous actions as “tokens“ by directly parameterizing the components of the GMM in each time step, avoiding the need for discretization and thus preserving the inherent geometry of the action space. This choice allows us to exploit the benefits of large-scale sequence modeling, such as autoregressive policy generation and access to explicit likelihood estimates, while avoiding the pitfalls of artificially segmenting continuous control signals into discrete bins. In addition, we discuss two different sampling strategies that reduce variability in the generated trajectories. In summary, our work: • Eliminates the need for action quantization in autoregressive policy parameterization, simplifying behavioral cloning pipelines while retaining the ability to directly evaluate action likelihoods. • Proposes two sampling algorithms to reduce the variance of the generated trajectories, including a global mode-finding algorithm. • Demonstrates state-of-the-art performance on standard benchmarks, both for conditional and unconditional policy generation using proprioceptive and image-based states, confirming the effectiveness of our method for multimodal policy modeling. The implementation is available at https://github.com/ziyadsheeba/qfat . (a) An overview of the Q-FAT architecture. (b) Sample trajectories generated by a Q-FAT policy. Figure 1 : Figure (a) shows the overview of QFAT. A sequence of h^{s} previous states and h^{g} goal states are projected into the Transformer’s embedding dimension using a linear layer. The Transformer then predicts the action distribution by predicting the GMM means, variances and mixture probabilities. Figure (b) shows a sample of generated trajectories from 3 representative environments, demonstrating the captured multi-modality in the sequence of solving tasks (Kitchen and UR3 Block Push) or the direction from which an object is approached (PushT). 2 Sequence to Action Prediction We consider the following standard setting for imitation learning of simulated robotics tasks. An agent equipped with continuous controls \mathcal{A} evolves in a continuous state space \mathcal{S} in discrete time. In behavioral cloning, we assume that a so-called behavior policy \pi_{b} collected a dataset D=\{\tau_{i}\}_{i=0}^{N-1} of N trajectories in the environment, where each trajectory \tau_{i}=\{s^{i}_{0},a^{i}_{0},\dots,a^{i}_{n^{i}-1},s^{i}_{n^{i}}\} is a sequence of states s^{i}_{t}\in\mathcal{S} and actions a^{i}_{t}\in\mathcal{A} , with n^{i} denoting the length of trajectory i . Note that we do not impose a Markovian assumption on the behavior policy. In general a^{i}_{t} might depend on a history of states \{s^{i}_{j}\}_{j=t-h}^{t} , where h is a fixed window. Our objective is to learn a sequence-to-action predictor, or imitating policy \pi_{\theta} . We treat the objective as a supervised learning problem, where the parameter \theta of our policy is optimized to minimize the negative log-likelihood with respect to \pi_{b} , \mathbb{E}_{\tau\sim\pi_{b}}[-\log(\pi_{\theta})(\tau)] . To do so, we minimize: \mathcal{L}(\theta)=-\frac{1}{N}\sum_{i=0}^{N-1}\sum_{t=0}^{n^{i}-1}\log\pi_{\theta}(a^{i}_{t}\mid s^{i}_{t-h:t})\,. (1) Note that a standard result in variational inference [ 15 ] shows that this objective is equivalent to minimizing the Kullback-Leibler (KL) divergence between the behavioral policy \pi_{b} and the learned policy \pi_{\theta} , implying that the learned policy \pi_{\theta} should match, or clone, the behavior of \pi_{b} . However, the choice of parametrization is crucial to guarantee a good approximation of the behavior policy. 3 The Q-FAT Algorithm Current transformer-based policy parametrization approaches [ 26 , 45 , 32 ] consist of first discretizing the action space, e.g. using a VQ-VAE [ 52 , 57 ] , to comply with the standard transformer architecture requiring a discrete vocabulary. Inspired by the success of GIVT [ 50 ] in the image generation domain, our model Quantization-Free Autoregressive Action Transformer (Q-FAT) adapts GIVT to the continuous action behavioral cloning setting, modeling the action distribution at each step by predicting the parameters of a multivariate Gaussian mixture model (GMM) . Namely, Q-FAT models the probability of an action a_{t}\in\mathbb{R}^{m} at step t as \pi_{\theta}\bigl(a_{t}\,\big|\;s_{t-1},\dots,s_{t-h^{s}})\!=\!\sum_{i=1}^{k}\pi_{i}^{t}\,\mathcal{N}\!\Bigl(a_{t}\;\big|\;\boldsymbol{\mu}_{i}^{t},\boldsymbol{\sigma}_{i}^{t}\Bigr)\,,\quad (2) where \pi_{i}^{t} , \boldsymbol{\mu}_{i}^{t} , and \boldsymbol{\mathbf{\sigma}_{i}^{t}} are produced by the transformer decoder [ 53 , 41 ] at step t , conditioned on a history of previous states \{s_{i}\}_{i=t-h^{s}}^{t-1} with s_{i}\in\mathbb{R}^{d}

Read the complete text