Adapting Interleaved Encoders with PPO for Language-Guided Reinforcement Learning in BabyAI
Aryan Mathur, Asaduddin Ahmed · 2025
Read Adapting Interleaved Encoders with PPO for Language-Guided Reinforcement Learning in BabyAI by Aryan Mathur, Asaduddin Ahmed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23148.txt.
Introduction
Aryan Mathur and Asaduddin Ahmed adapt an interleaved encoder with PPO training for language-guided tasks in the BabyAI world. Agents often split seeing and deciding into separate modules, so the policy cannot teach perception what matters. The PDiT design alternates perception and decision layers inside one transformer, which the authors implement here.
They train the interleaved model with proximal policy optimization on tasks that pair vision with language instructions. Tight coupling lets decision errors shape perception directly. The study tests whether that shared learning helps small agents follow spoken goals in grid worlds more reliably.
Catalog record
| CREATOR | Aryan Mathur, Asaduddin Ahmed |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23148
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adapting Interleaved Encoders with PPO for Language-Guided Reinforcement Learning in BabyAI Aryan Mathur, Asaduddin Ahmed Abstract Deep reinforcement learning agents often struggle when tasks require understanding both vision and language. Conventional architectures typically isolate perception (e.g., CNN-based visual encoders) from decision-making (policy networks). This separation can be inefficient, as the policy’s failures don’t directly help the perception module learn what’s important. To address this, we implement the Perception–Decision Interleaving Transformer (PDiT) architecture, introduced by Mao et al. (2023), a model that alternates between perception and decision layers within a single transformer. This allows feedback from decision-making to directly refine perceptual features. Additionally, we integrate a contrastive loss inspired by CLIP to align textual mission embeddings with visual scene features. We evaluate PDiT on the BabyAI GoToLocal environment, demonstrating it achieves more stable rewards and better alignment compared to a standard PPO baseline. The results suggest interleaved architectures are a promising direction for more integrated autonomous agents. I Introduction In reinforcement learning (RL), perception and policy are often treated as separate problems. A vision module, like a CNN, encodes the state, and a policy network then decides on an action. This separation is a missed opportunity, as the policy’s failures don’t directly help the perception module learn what’s important. This limitation becomes clear in tasks requiring language. For an agent to follow ”Go to the red ball,” it must learn to *look for* the red ball. This visual ”looking” should ideally be shaped by its past successes or failures in finding that ball. Architectures that interleave perception and decision-making have been proposed as a way to create this tighter feedback loop [ 1 ] . By alternating perception and policy layers, the model can continuously refine its understanding of the state. However, models like Vision Transformers (ViT) [ 2 ] and Decision Transformers (DT) [ 3 ] still largely treat perception and control as separate sequential stages. Motivation. While interleaved architectures are promising, their effectiveness in sparse-reward, language-guided tasks like BabyAI is an open question. We hypothesize that simply applying such an encoder is not enough; it must be tightly integrated with the policy and given a strong multimodal signal. We aim to investigate if combining an interleaved encoder with a PPO policy and a CLIP-style contrastive loss can improve stability and sample efficiency. I-A Problem Statement Given a multimodal environment E=(S,A,P,R) , with states s_{t}\in S consisting of visual observations o_{t} and textual missions m_{t} , our goal is to learn a policy \pi(a_{t}|s_{t},m_{t}) that maximizes expected cumulative reward: J(\pi)=\mathbb{E}_{\pi}\Big[\sum_{t=0}^{T}\gamma^{t}r_{t}\Big]. Instead of encoding s_{t} just once, PDiT continuously refines its understanding of the state throughout the decision-making process. I-B Contributions Based on this, our work makes the following contributions: • We adapt and apply an interleaved perception-decision transformer encoder to the BabyAI GoToLocal environment, integrating it with a PPO policy. • We propose a joint optimization objective that combines the PPO loss with a CLIP-style contrastive loss to improve visual-textual grounding. • We empirically evaluate this combined framework, demonstrating it achieves more stable convergence and lower reward variance than a standard PPO baseline. • We provide an analysis of the benefits of interleaving and contrastive alignment for this specific task. II Background and Related Work II-A Reinforcement Learning Fundamentals In policy-gradient RL, the objective is to optimize parameters \theta to maximize: J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t}\gamma^{t}r_{t}\right], with gradients estimated as: \nabla_{\theta}J(\theta)=\mathbb{E}\big[\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\hat{A}_{t}\big], where \hat{A}_{t} is the advantage estimator. PPO stabilizes this update by clipping the ratio r_{t}(\theta) : \mathcal{L}_{PPO}=\mathbb{E}_{t}\Big[\min(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})\Big]. (1) This prevents large policy updates that destabilize learning. II-B Transformers in Perception and Decision-Making The transformer’s self-attention mechanism, defined as: \text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V, provides a context-aware representation that can model long-range dependencies. Vision Transformers (ViT) apply this concept to image patches, while Decision Transformers use it to represent trajectory histories. II-C Contrastive Multimodal Learning CLIP [ 4 ] aligns image and text pairs using symmetric InfoNCE loss: \displaystyle\mathcal{L}_{CLIP} \displaystyle=-\frac{1}{N}\sum_{i}\log\frac{\exp(\text{sim}(v_{i},t_{i})/\tau)}{\sum_{j}\exp(\text{sim}(v_{i},t_{j})/\tau)}. (2) This loss pushes corresponding visual and textual embeddings closer in latent space, improving cross-modal understanding. PDiT borrows this principle to bind visual frames from BabyAI with mission instructions, ensuring alignment between perception and semantic command understanding. III Proposed Method: PDiT Framework III-A Architecture Overview The PDiT framework integrates two alternating stacks: PDiT=[P_{1},D_{1},P_{2},D_{2},\ldots,P_{L},D_{L}], where P_{l} is a perception transformer and D_{l} a decision transformer layer. Each P_{l} encodes the multimodal state (s_{t},m_{t}) , while D_{l} generates an updated policy embedding conditioned on P_{l} ’s output. Fig. 1: Conceptual diagram of Perception–Decision Interleaving Transformer showing visual-text input fusion, interleaved layers, and joint optimization paths. III-B Mathematical Formulation At each timestep t , we define: \displaystyle x_{t} \displaystyle=f_{v}(o_{t}),\quad y_{t}=f_{t}(m_{t}), (3) \displaystyle z_{t} \displaystyle=P([x_{t};y_{t};a_{t-1};r_{t-1}]), (4) \displaystyle\pi(a_{t}|s_{t}) \displaystyle=D([z_{t};r_{t};a_{t-1}]). (5) The transformer alternates between updating perceptual latent z_{t} and computing policy logits for a_{t} . The total loss is: \mathcal{L}_{total}=\mathcal{L}_{PPO}+\lambda_{1}\mathcal{L}_{CLIP}+\lambda_{2}\mathcal{L}_{sup}, (6) where \mathcal{L}_{sup} denotes imitation learning supervision loss. III-C Joint Gradient Coupling Gradient coupling between modules can be expressed as: \nabla_{\theta_{P}}\mathcal{L}_{total}=\nabla_{\theta_{P}}(\mathcal{L}_{CLIP})+\alpha\nabla_{\theta_{P}}(\mathcal{L}_{PPO}), enabling perception layers to evolve with policy learning signals — unlike frozen encoders in standard RL architectures. IV System Design and Implementation IV-A Environment Setup We evaluate PDiT in the BabyAI GoToLocal-V0 environment [ 6 ] , a partially observable 8×8 grid where the agent (a red triangle) must navigate to an object of specified color and type. The mission is provided as natural language (e.g., “Go to the green key”). Each step yields a binary reward (1 on success, 0 otherwise). Fig. 2: BabyAI GoToLocal environment. The agent perceives a 7×7 field of view and receives mission instructions as text. IV-B Training Parameters • PPO: learning rate = 3\times 10^{-4} , \gamma=0.99 , steps=256, batch size=64 • PDiT: hidden dim = 64, kernel size = 3, stride = 1, padding = 1 • Supervised: embedding dim=128, heads=2, batch=512, LR= 1\times 10^{-4} IV-C Contrastive Alignment Implementation We used a pre-trained CLIP encoder to project image and text tokens into a shared space. During training, the InfoNCE loss encourages: \text{sim}(f_{v}(o_{t}),f_{t}(m_{t}))>\text{sim}(f_{v}(o_{t}),f_{t}(m_{j})), for j\neq t , ensuring semantic consistency. V Experimental Results and Analysis V-A Quantitative Results Metric Value (PDiT) Mean Episode Reward 0.27 \pm 0.36 Convergence Steps 160k Entropy Coefficient 0.004 Reward Variance Reduction 42% TABLE I: Performance of PDiT–PPO vs baseline PPO on BabyAI. V-B Qualitative Behavior The interleaving mechanism produced smoother navigation trajectories. Visualizations of attention weights indicated strong alignment between mission tokens (“red ball”) and corresponding visual patches. Agents trained without CLIP alignment often fixated on distractor objects. V-C Policy Stability Metric Define stability ratio \mathcal{S} : \mathcal{S}=\frac{\text{Var}(R_{t}^{PPO})}{\text{Var}(R_{t}^{PDiT})}. We obtained \mathcal{S}=1.73 , indicating a 73% improvement in policy smoothness across episodes. VI Theoretical Justification VI-A Why Interleaving Works Traditional RL architectures assume: z_{t}=f(s_{t})\quad\text{and}\quad a_{t}=\pi(z_{t}). Thus, \nabla a_{t}/\nabla s_{t} is indirect. In PDiT: a_{t}^{(l)}=D_{l}(P_{l}(s_{t},a_{t-1})), so the gradient of policy w.r.t. raw state becomes: \frac{\partial a_{t}}{\partial s_{t}}=\frac{\partial D_{l}}{\partial P_{l}}\frac{\partial P_{l}}{\partial s_{t}}, implying direct gradient flow between policy and perception. This forms an implicit bi-level optimization loop akin to meta-learning. VI-B Complexity Analysis Given L interleaved layers, computational cost scales as \mathcal{O}(Ln^{2}d) for n tokens of dimension d . Despite this, interleaving yields a higher gradient signal-to-noise ratio, justifying the added overhead. VII Discussion VII-A Interpretation of Results Interleaving perception and decision reasoning appears to enhance feature relevance. The perception modules get implicit supervision from reward gradients, forcing them to refine representations to be