From Abstraction to Reality: DARPA’s Vision for Robust Sim-to-Real Autonomy

Erfaun Noorani, Zachary Serlin, Ben Price, Alvaro Velasquez · 2025

Read From Abstraction to Reality: DARPA’s Vision for Robust Sim-to-Real Autonomy by Erfaun Noorani, Zachary Serlin, Ben Price, Alvaro Velasquez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11007.txt.

Introduction

Erfaun Noorani, Zachary Serlin, Ben Price, and Alvaro Velasquez lay out a DARPA program in a 2025 paper on self-driving systems. Robots and vehicles are often trained in simulators, then fail when the real world differs from the sim. Most fixes try to make the sim ever more exact, or assume the gap is small.

The program they describe, called TIAMAT, takes the other path. It trains systems across many rough, low-detail sims that share the same basic meaning, so the learned skills rest on shared sense rather than fine detail. The aim is fast, robust transfer to real settings that are complex, shifting, and time-pressured. The paper sets out the plan and the hurdles in reshaping the full autonomy pipeline around it.

Catalog record

CREATORErfaun Noorani, Zachary Serlin, Ben Price, Alvaro Velasquez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11007

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Abstraction to Reality: DARPA’s Vision for Robust Sim-to-Real Autonomy Erfaun Noorani, Zachary Serlin, Ben Price, Alvaro Velasquez Abstract The DARPA T ransfer from I mprecise and A bstract M odels to A utonomous T echnologies (TIAMAT) program aims to address rapid and robust transfer of autonomy technologies across dynamic and complex environments, goals, and platforms. Existing methods for simulation-to-reality (sim-to-real) transfer often rely on high-fidelity simulations and struggle with broad adaptation, particularly in time-sensitive scenarios. Although many approaches have shown incredible performance at specific tasks, most techniques fall short when posed with unforeseen, complex, and dynamic real-world scenarios due to the inherent limitations of simulation. In contrast to current research that aims to bridge the gap between simulation environments and the real world through increasingly sophisticated simulations and a combination of methods typically assuming a small sim-to-real gap—such as domain randomization, domain adaptation, imitation learning, meta-learning, policy distillation, and dynamic optimization—TIAMAT takes a different approach by instead emphasizing transfer and adaptation of the autonomy stack directly to real-world environments by utilizing a breadth of low(er)-fidelity simulations to create broadly effective sim-to-real transfers. By abstractly learning from multiple simulation environments in reference to their shared semantics, TIAMAT’s approaches aim to achieve abstract-to-real transfer for effective and rapid real-world adaptation. Furthermore, this program endeavors to improve the overall autonomy pipeline by addressing the inherent challenges in translating simulated behaviors into effective real-world performance. † † footnotetext: Erfaun Noorani, Zachary Serlin, and Ben Price are with MIT Lincoln Laboratory (emails: {erfaun.noorani, zachary.serlin, ben.price}@ll.mit.edu ). Alvaro Velasquez is with Defense Advanced Research Projects Agency (DARPA) (email: {alvaro.velasquez}@darpa.mil). DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering. This work has been submitted for publication in AI Magazine. Index terms— DARPA, TIAMAT, Abstract-to-real, sim-to-sim, sim-to-real, adaptation, adaptive, robust autonomy, transfer 1 Introduction and Motivation Humans possess adaptive expertise—the ability to quickly apply knowledge, plans, and strategies learned in simpler contexts to more complex and unfamiliar situations. An example of adaptive expertise is a firefighter responding to an emergency situation. Although firefighters are trained in various standardized procedures and protocols, real-world scenarios often present unpredictable challenges, such as unusual weather conditions, changing fire patterns, or obstructions in the environment. An experienced firefighter draws on their past knowledge and skills, adapting their approach based on the specific situation at hand. For example, if a firefighter arrives at a fire that has spread rapidly due to high winds, they might adjust their strategy—such as prioritizing certain areas for control or choosing a different approach to ventilation—based on the conditions they face, even if those conditions weren’t part of their standard training. The fundamental goal of the TIAMAT program is to enable autonomous systems to develop similar adaptive expertise—quickly creating and executing effective plans based on minimal prior knowledge and generalizing strategies from previous tasks, possibly learned in less complex and more controlled environments, in unpredictable environments with minimal adaptation time. This capability is crucial for deploying autonomous systems in dynamic, real-world environments where they must handle complex, unforeseen challenges. Understanding the Simulation to Reality (sim-to-real) Gap: One foundational hurdle in advancing fielded autonomous systems is translating the success achieved in simulations into practical, real-world situations. The sim-to-real gap refers to the difference between how autonomous systems perform in simulations compared to the real world. While simulations are valuable for training, they cannot fully replicate the dynamic, complex and unpredictable nature of real-world environments. This gap arises because factors such as environmental variability, sensory noise, and complex dynamics are difficult to simulate accurately. While there have been some initial attempts to quantify these gaps, such as through the sim-to-real correlation coefficient [ 1 ] , they are still mostly notional. To understand how autonomous systems navigate this challenge, we need to examine the key components that these systems rely on: perception, planning, and control. Perception is how the system senses and interprets its surroundings, such as identifying objects or detecting obstacles. Planning is the process through which the system determines its course of action, like selecting the optimal route or strategy. Control is how the system executes those actions, such as steering, accelerating, or braking. The sim-to-real gap affects each of these dimensions: in perception, simulated sensors may not capture the full range of real-world conditions, such as lighting or weather; in planning, virtual scenarios are often more predictable, leading to overconfidence in decision-making; and in control, real-world dynamics such as tire friction or road surface variations can introduce unexpected behaviors. Together, these gaps make it difficult for systems to seamlessly transition from simulation to real-world deployment. In our view, bridging sim-to-real gaps can be broken down into the above three types of transfer (perception, planning, and control). • Perception Transfer: This refers to a system’s ability to interpret sensory data and semantics in real-world conditions compared to simulations. For instance, a robot trained in a virtual environment might struggle to recognize objects under different lighting, weather conditions, or when those objects are partially damaged, broken, or hidden. A gap in perception transfer could manifest as the robot misidentifying a road sign due to variations in color or shape. • Planning Transfer: This involves a system’s capacity to apply strategies to navigate and make decisions in real-world scenarios. For example, a simulated robot may effectively plan a route in an ideal environment but struggle to adapt when faced with unexpected obstacles or noise from a perception system. A gap in planning transfer might be evident if the robot has difficulty rerouting when encountering a previously unknown barrier or must replan when a route is deemed impassable. • Control Transfer: This concerns how effectively a system executes commands in real-world conditions. For example, a quadruped robot that runs smoothly in simulation might struggle with stability in muddy conditions. A gap in control transfer could become apparent if the drone behaves erratically or fails to maintain its intended path when faced with real-world turbulence. Even though we have broken down the transfer into these three components, it is important to recognize that the autonomy stack functions as a system of systems. Its overall performance and transferability depend not only on how well each individual component adapts but also on how effectively these components interact across the entire stack. The integration and seamless functioning of perception, planning, and control are crucial for ensuring that the system operates efficiently in real-world scenarios. Therefore, addressing the sim-to-real gap requires a holistic approach that considers the inter-dependencies among all elements of the autonomy stack. The Limitations of Better Simulations and existing methods: Efforts to develop more sophisticated and realistic simulations have undoubtedly contributed to advancements in autonomous systems. Yet, even the most advanced simulations fall short in replicating the full spectrum of real-world conditions. Additionally, existing algorithms, such as domain randomization [ 2 ] , domain adaptation [ 3 ] , imitation learning [ 4 ] , meta-learning [

Read the complete text