DISC: Decoupling Instruction from State-Conditioned Control via Policy Generation * Equal contribution. † Corresponding author.

Hanxiang Ren ‡,§,¶,* Pei Zhou §,¶,* Xunzhe Zhou §,¶ Yanchao Yang §,¶,† · 2026

Read DISC: Decoupling Instruction from State-Conditioned Control via Policy Generation * Equal contribution. † Corresponding author. by Hanxiang Ren ‡,§,¶,* Pei Zhou §,¶,* Xunzhe Zhou §,¶ Yanchao Yang §,¶,† (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20856.txt.

Introduction

Hanxiang Ren and three colleagues present DISC, a method that splits instruction from state-conditioned control via policy generation. Language-conditioned policies usually share network parameters for instructions and observations. That lets networks find scene-to-action shortcuts that skip language grounding. DISC instead uses a hypernetwork to generate a full task-specific visuomotor policy from the instruction alone, so observation leakage has no pathway to emerge.

Catalog record

CREATORHanxiang Ren ‡,§,¶,* Pei Zhou §,¶,* Xunzhe Zhou §,¶ Yanchao Yang §,¶,†
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92158
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20856

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DISC: Decoupling Instruction from State-Conditioned Control via Policy Generation * Equal contribution. † Corresponding author. Hanxiang Ren ‡,§,¶,* Pei Zhou §,¶,* Xunzhe Zhou §,¶ Yanchao Yang §,¶,† Abstract Language-conditioned manipulation policies typically process instructions and observations through shared network parameters. This task-state entanglement provides a pathway for observation leakage – networks learn scene-to-action shortcuts that bypass language grounding entirely. DISC eliminates this failure structurally. Rather than conditioning a universal policy on language, DISC uses a hypernetwork to generate the entire parameter set of a task-specific visuomotor policy from the instruction alone. The generated policy never directly accesses language; therefore, its task-awareness must come from the language. Consequently, observation leakage has no pathway to emerge. On the other hand, generating coherent high-dimensional policy weights is itself a challenging problem. We address it with a two-stage hypernetwork whose refinement stage embeds the structure of gradient-based optimization as a feed-forward inductive bias, producing globally consistent parameters without actual gradient computation. Trained entirely from scratch on standard data budgets, DISC outperforms all entangled baselines on LIBERO-90 and Meta-World, with advantages that widen on complex, long-horizon tasks – and surpasses the large-scale pretrained π 0 \pi_{0} despite using no external pretraining data. On a real-world benchmark where all tasks share identical visual context, DISC substantially outperforms entangled alternatives, directly confirming that language-generated policy parameters, not visual shortcuts, drive behavior. The hypernetwork further learns a semantically structured parameter manifold that enables few-shot adaptation from minimal demonstrations and robust generalization across paraphrased instructions. Our code is available at: https://github.com/ReNginx/DISC . I Introduction Reliable task-conditioned manipulation requires robots to treat the task specification as the source of task identity, rather than as a weak hint that can be overridden by familiar visual scenes. Language-conditioned manipulation policies must process two fundamentally different inputs: an instruction that specifies what task to perform (fixed throughout an episode) and an observation that captures what the current state is (evolving continuously). Current architectures conflate these roles by processing both through shared representations – a design we term task-state entanglement . We argue this creates two compounding failures. First, shared parameters face optimization tension between language grounding (requiring semantic understanding) and visuomotor control (requiring spatial precision). Second, and more critically, entangled architectures enable observation leakage : visual context often correlates with task identity in limited robotics datasets, allowing the network to map scenes directly to actions while bypassing language entirely. For example, in a shared kitchen scene, a policy may learn the frequent pan-placement behavior and omit the instructed stove-activation subgoal. The result is policies that exploit visual shortcuts rather than ground instructions – succeeding on familiar scenes but failing when instructions change or visual context becomes ambiguous. Evidence from the broader vision-language community confirms this pattern, manifesting as modality competition [ 22 , 40 ] , visual neglect [ 24 ] , and entity grounding failures [ 1 , 25 ] . We propose DISC , which takes architectural decoupling to its logical conclusion: a task specification, instantiated here as language, generates the entire parameter set of a task-specific visuomotor policy. A hypernetwork processes instructions alone to produce policy weights; the generated policy processes visual observations alone using those weights. This separation provides a structural guarantee against observation leakage – no task-agnostic parameters exist where shortcuts could emerge, and task semantics influence behavior only through generated weights. A central challenge is that naive weight generation produces incoherent parameters. DISC addresses this with a two-stage architecture. A Weight Initialization Network first maps the language embedding to a semantically informed point in parameter space. An Iterative Refinement Module then improves this initialization by mimicking the structure of gradient-based optimization – forward evaluation, error estimation, and parameter correction – while remaining fully feed-forward at inference time. This optimization-inspired inductive bias enables the hypernetwork to produce globally coherent policy parameters without incurring the cost of actual gradient computation, bridging the gap between the expressiveness of full policy generation and the quality demands of precise visuomotor control. Despite training entirely from scratch, DISC achieves 94.3% on LIBERO-90 and 92.2% on Meta-World, outperforming the strongest trained-from-scratch entangled baseline by 7.7% on LIBERO-90. For context, DISC surpasses even the pretrained \pi_{0} (91.6%) and remains competitive with \pi_{0.5} (95.7%) – demonstrating that architectural inductive biases can recover much of the benefit attributed to large-scale pretraining. The advantage widens on complex, long-horizon tasks, precisely where task-state entanglement is most damaging. On a real-world combinatorial benchmark where visual context is shared across all 9 tasks, DISC achieves 86.4% versus 78.5% for the best entangled baseline, confirming that task-specific generated parameters, not visual shortcuts, drive behavior. The decoupled architecture further enables superior few-shot adaptation, robust grounding across paraphrased instructions, and a semantically structured parameter manifold where related tasks cluster closely by functional meaning. II Related Work Multi-Task Manipulation Policies. Current language-conditioned manipulation methods process instructions and observations through shared network components and differ mainly in their fusion mechanism. Transformer-based approaches [ 5 , 17 , 45 , 49 , 47 ] tokenize and concatenate multimodal inputs into unified sequences [ 42 ] , as exemplified by RT-1 [ 4 ] , Gato [ 30 ] , Octo [ 41 ] , VQ-BeT [ 37 ] , and BAKU [ 10 ] – where the same W_{Q} , W_{K} , W_{V} projections and feed-forward weights operate on both language and visual tokens. Text-aware variants such as OTTER [ 13 ] condition visual features on language, but retain shared action-prediction parameters. Concept-guided policies such as AutoCGP [ 46 ] learn semantic control structure from unlabeled demonstrations. Diffusion-based methods [ 11 , 39 ] instead cast control as conditional denoising, pioneered by Diffusion Policy [ 6 ] . Architectures have evolved from U-Nets [ 35 ] with FiLM conditioning [ 27 ] to Transformer-based backbones such as DiT [ 26 ] and BESO [ 32 ] , with RDT-1B [ 21 ] and MoDE [ 33 ] scaling through pre-training and mixture-of-experts tuning, while DP3 [ 44 ] , 3D Diffuser Actor [ 16 ] , and PointVLA [ 18 ] enhance geometric grounding via 3D representations. Flow-matching models \pi_{0} [ 3 ] and \pi_{0.5} [ 15 ] further improve generalization through large-scale pre-training. Despite this diversity, all these methods entangle instruction and state within shared processing stages. Concatenation-based transformers apply identical weights to both modalities throughout the network. Cross-attention and FiLM-based architectures [ 27 ] introduce partial separation – FiLM affects vision only through affine modulation – but language still controls only a small fraction of computation, while the vast majority of parameters remain task-agnostic and can learn shortcut mappings that bypass language entirely. As analyzed in Section III-A , this task-state entanglement creates two problems: optimization tension between competing functions within shared parameters, and observation leakage that enables the network to map visual context directly to actions without grounding language. Hypernetworks for Policy Generation. An alternative paradigm uses hypernetworks [ 2 , 9 , 14 , 48 ] – networks trained to output the weights of another – to generate task-specific policies. HyPoGen [ 31 ] , HyperZero [ 34 ] , and Hyper-GoalNet [ 48 ] demonstrate this for multi-task or goal-conditioned manipulation. These methods motivate separating task specification from policy execution, but differ

Read the complete text