Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternatives
Chengyang He, Xu Liu, Gadiel Sznaier Camps, Guillaume Sartoretti, Mac Schwager · 2025
Read Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternatives by Chengyang He, Xu Liu, Gadiel Sznaier Camps, Guillaume Sartoretti, Mac Schwager (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05787.txt.
Introduction
Chengyang He and co-authors argue that diffusion policies for robots mostly memorize actions in a 2025 paper. Such policies show striking dexterity on hard manipulation tasks while training on few demonstrations, but why they work was a mystery. Their hypothesis is that at run time the policy finds the closest training image in a latent space and recalls its action sequence, rather than generalizing.
Evidence backs the claim, since even wild out of range images of cats and dogs still trigger a training action sequence. That lookup can be a virtue when data are sparse. The authors build a simple lookup table policy on that insight, using a contrastive image encoder as an index, as a light alternative.
Catalog record
| CREATOR | Chengyang He, Xu Liu, Gadiel Sznaier Camps, Guillaume Sartoretti, Mac Schwager |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05787
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternatives Chengyang He, Xu Liu, Gadiel Sznaier Camps, Guillaume Sartoretti, Mac Schwager Abstract Diffusion policies have demonstrated remarkable dexterity and robustness in intricate, high-dimensional robot manipulation tasks, while training from a small number of demonstrations. However, the reason for this performance remains a mystery. In this paper, we offer a surprising hypothesis: diffusion policies essentially memorize an action lookup table— and this is beneficial . We posit that, at runtime, diffusion policies find the closest training image to the test image in a latent space, and recall the associated training action sequence, offering reactivity without the need for action generalization. This is effective in the sparse data regime, where there is not enough data density for the model to learn action generalization. We support this claim with systematic empirical evidence. Even when conditioned on wildly out of distribution (OOD) images of cats and dogs, the Diffusion Policy still outputs an action sequence from the training data. With this insight, we propose a simple policy, the Action Lookup Table (ALT), as a lightweight alternative to the Diffusion Policy. Our ALT policy uses a contrastive image encoder as a hash function to index the closest corresponding training action sequence, explicitly performing the computation that the Diffusion Policy implicitly learns. We show empirically that for relatively small datasets, ALT matches the performance of a diffusion model, while requiring only 0.0034 of the inference time and 0.0085 of the memory footprint, allowing for much faster closed-loop inference with resource constrained robots. We also train our ALT policy to give an explicit OOD flag when the distance between the runtime image is too far in the latent space from the training images, giving a simple but effective runtime monitor. More information can be found at: https://stanfordmsl.github.io/alt/ . Keywords: Robot Manipulation, Diffusion Policy, Action Memorization 1 Introduction Imitation learning for robot manipulation requires training a policy to map from image inputs to action sequence outputs given a relatively small number of demonstrations. Recently, the Diffusion Policy [ 1 ] has emerged as a powerful and novel approach to this problem by modeling the robot’s visuomotor policy as a denoising diffusion probabilistic model [ 2 ] . Diffusion models are generative models that train a filter to iteratively remove noise from noise-corrupted training data. At inference time a random output is sampled, and progressively denoised (conditioned on the input) to produce an inference. The diffusion model was originally introduced for image generation and remains the dominant architecture in that domain [ 3 , 4 ] . The primary advantage of the Diffusion Policy for robot maipulation lies in its ability to model multi-modal action distributions, scale to high-dimensional outputs, and produce long-horizon action sequences. Indeed, recent studies have shown that diffusion policies outperform many existing methods on challenging manipulation benchmarks [ 5 , 6 ] . The performance of the Diffusion Policy is unquestionable, however the explanation for this performance remains elusive. In particular, typical diffusion policies are trained on 50–200 task demonstrations (a small amount of data), while maintaining the same number of parameters (typically over 100 million) as image generation models trained on billions of images [ 3 , 4 ] . Furthermore, the common practice is to train Diffusion Policies until the training loss is low, but the test loss is high—the classic signal for over-fitting in machine learning. In typical machine learning, overfitting is associated with poor test-time performance and poor generalization. Yet it is observed that this overfitting is actually necessary for strong test time performance of the Diffusion Policy. The question arises: Why do diffusion policies trained to overfit small data sets appear to give strong test-time performance in robot manipulation? In this paper we show that, indeed, diffusion policies severely overfit the training data, such that they memorize the training action sequences. At inference, diffusion policies recall the training action sequences with nearly no generalization. They essentially perform a lookup table that maps runtime images to training action sequences. Combined with online closed-loop execution with runtime images, this action memorization appears to be a winning recipe for strong manipulation policies obtained from small amounts of demonstration data. The stochastic multi-modality of the policy results from slight randomness in which training action sequence is indexed, not from generalization over action sequences. However, one key drawback of the Diffusion Policy is slow inference time, which leads to slow robot execution punctuated by pauses as the model recomputes the next inference at the end of each action sequence. This motivates our second research question: Can the same action memorization behavior be accomplished with a simpler, faster model architecture to yield faster runtime performance? To answer this question, we propose a simple lookup table policy with a trained image encoder to map from images to actions, which we call the Action Lookup Table (ALT) policy, illustrated in Fig. 1 . The ALT policy performs similarly to the Diffusion Policy, while being 300 times faster at inference, and requiring less than 1/100th the memory footprint. Figure 1: Resulting latent space of our constrastive learning (CL) based ALT on our training data, illustrating the distribution of training examples and example in- (1,2) and out-of-distribution (3) test images. Concretely, we design a lightweight image-joint pose encoder that maps each provided observation into a low-dimensional feature representation (LFR), which we then use to create a memory bank of LFRs from the training expert demonstrations. During inference, we then encode new observations, find the nearest code from the training set in the latent space, and output the memorized training action sequence associated with that code. The enccoder is trained using a contrastive learning objective [ 7 ] , encouraging positive sample pairs to be embedded closer together while pushing negative pairs apart in the latent space. ALT avoids the costly iterative denoising steps, enabling faster (single forward-pass) inference. Our contributions are as follows: (1) We hypothesize the Diffusion Policy implicitly memorizes an action lookup table. We provide conceptual intuition for this hypothesis and support it through extensive empirical validation. (2) We present a new ALT policy (as shown in Fig. 1 ), a contrastive learning-based alternative to diffusion policies, explicitly implementing an action sequence lookup table by encoding image-joint pose inputs into a latent space to index training action sequences. By setting a threshold distance in the latent space, we obtain a simple OOD detector to flag potential policy failures at runtime. 2 Prior Work Diffusion models, trained by gradually adding Gaussian noise to data during training [ 2 , 8 , 9 ] , were originally developed for high-dimensional data generation tasks such as image, video, or audio synthesis [ 10 , 11 ] . These models can produce seemingly novel high-quality images and videos in a variety of different styles through simple text [ 8 , 12 , 13 , 3 ] and image [ 14 , 15 , 16 ] prompt conditioning. In order to capture the complex multimodal distributions inherent in visual and auditory data, these models are often large, containing from hundreds of millions to billions of parameters [ 3 , 4 ] , and are trained over large datasets with hundreds of millions to