Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer

Kevin Qiu, Kyle Walker, Mike Y. Michelis, Marek Cygan, Josie Hughes · 2026

Read Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer by Kevin Qiu, Kyle Walker, Mike Y. Michelis, Marek Cygan, Josie Hughes (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20827.txt.

Introduction

Qiu and co-authors present Swim2Real, a pipeline that tunes a robotic fish simulator from swimming videos. A vision language model compares real and simulated clips. It then proposes updates to all 16 simulator parameters at once. A backtracking line search checks each step size. This tripled the accept rate, from 14 to 42 percent. The tuned model matched real fish speeds best across motor frequencies. Policies trained in it swam 12 percent farther than BayesOpt tuned ones. Commands then ran on the real fish at 50 hertz.

Catalog record

CREATORKevin Qiu, Kyle Walker, Mike Y. Michelis, Marek Cygan, Josie Hughes
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33723
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20827

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer Kevin Qiu, Kyle Walker, Mike Y. Michelis, Marek Cygan, Josie Hughes Abstract We present Swim2Real, a pipeline that calibrates a 16-parameter robotic fish simulator from swimming videos using vision-language model (VLM) feedback, requiring no hand-designed search stages. Calibrating soft aquatic robots is particularly challenging because nonlinear fluid-structure coupling makes the parameter landscape chaotic, simplified fluid models introduce a persistent sim-to-real gap, and controlled aquatic experiments are difficult to reproduce. Prior work on this platform required three manually tailored stages to handle this complexity. The VLM compares simulated and real videos and proposes parameter updates. A backtracking line search then validates each step size, tripling the accept rate from 14% to 42% by recovering proposals where the direction is correct but the magnitude is too large. Swim2Real calibrates all 16 parameters simultaneously, most closely matching real fish velocities across all motor frequencies (MAE = 7.4 mm/s, 43% lower than the next-best method), with zero outlier seeds across five runs. Motor commands from the trained policy transfer to the physical fish at 50 Hz, completing the pipeline from swimming video to real-world deployment. Downstream RL policies swim 12% farther than those from BayesOpt-calibrated simulators and 90% farther than CMA-ES. These results demonstrate that VLM-guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification, a step toward automated, general-purpose simulator tuning for underwater robotics. I INTRODUCTION Sim-to-real transfer requires simulators that accurately reproduce real-world dynamics [ 20 ] . Calibrating simulators for rigid-body systems benefits from analytical models and domain randomization [ 29 , 23 ] , but for soft and aquatic robots this remains challenging due to nonlinear material responses, coupled parameter spaces and fluid-structure interactions that are difficult to tune manually or through standard optimization techniques [ 4 , 18 ] . Ideally, calibration would be fully automatic, requiring only video of the real robot and producing calibrated simulator parameters with no manual intervention. This removes the need for the practitioner to decide which parameters to tune, which methods to use, or in what order. Any inaccurate calibration directly degrades downstream policy performance (Sec. V ). System identification fits simulator parameters to match observed behavior and is the classical approach to bridging this gap [ 15 ] . Recent methods frame system identification as a black-box process, applying Bayesian optimization [ 27 ] or evolutionary strategies [ 9 ] to search parameter spaces. These methods treat the simulator as an opaque function and rely solely on scalar error signals, ignoring the rich visual and physical structure of the comparison. Recent work has demonstrated that video comparisons between simulated and real robots can drive system identification without hand-crafted cost functions, exploiting the visual structure that scalar metrics discard [ 26 ] . Vision-language models (VLMs) such as Gemini [ 7 ] provide a structured diagnostic signal. Rather than returning a scalar error, they reason about physical discrepancies from video, identifying when a simulated fish bends too sharply, moves too slowly, or exhibits incorrect tail dynamics. This VLM-based physical reasoning complements traditional optimizers, proposing directions informed by physical intuition, and a line search determines the correct magnitude . To this end, we present Swim2Real, a VLM-guided system identification pipeline that takes raw swimming videos as input and automatically calibrates all 16 parameters of a tendon-driven fish simulator in MuJoCo [ 30 ] . Prior work on this platform [ 19 ] calibrated 9 parameters in three manually decomposed stages (stiffness via FFT analysis, motor geometry via grid search, and fluid coefficients via Bayesian optimization), requiring the practitioner to decide which parameters to group, which method to apply to each group, and in what order. Swim2Real calibrates all 16 parameters simultaneously (fluid coefficients, motor geometry, and per-joint stiffness and damping) with no hand-designed search stages, by iteratively comparing simulated and real swimming videos across eight actuation frequencies. A backtracking line search [ 21 ] validates each VLM proposal at geometrically decreasing step sizes, recovering updates where the VLM direction is correct but the magnitude is too large. The contributions of this work are as follows: 1. Swim2Real, an end-to-end pipeline that calibrates all 16 simulator parameters directly from swimming videos, with no hand-designed search stages or per-parameter method selection. 2. Comprehensive sim-to-real validation: the calibrated simulator most closely matches real fish velocities across all motor frequencies (MAE = 7.4 mm/s, 43% lower than the next-best method), with zero outlier seeds across five runs. 3. Real-world deployment of RL policies trained in the calibrated simulator, which swim 12% farther than BayesOpt-calibrated policies, completing the pipeline from video input to physical robot execution. II RELATED WORK Sim-to-Real Transfer. Domain randomization [ 29 , 23 ] and its adaptive variants [ 3 ] train policies that are robust to simulator inaccuracies by sampling parameter distributions. While effective for rigid-body tasks [ 10 ] , these methods do not reduce the sim-to-real gap itself but instead train around it. System identification offers a complementary approach by calibrating the simulator so that a single accurate model suffices for transfer [ 32 ] . System Identification for Robotics. Classical system identification fits parametric models to input-output data [ 15 ] . In the sim-to-real setting, BayesSim [ 27 ] estimates parameter posteriors via likelihood-free inference, while black-box methods apply Bayesian optimization [ 28 ] or CMA-ES [ 9 ] to minimize trajectory-matching errors [ 5 ] . These approaches rely on scalar cost functions and discard the visual structure of the comparison. Differentiable simulation [ 11 ] provides analytical gradients for system identification, including for soft robotic fish [ 33 , 6 ] , but the stateless ellipsoid fluid model used here lacks differentiable backend support. For soft robotic fish [ 18 , 12 ] , coupled fluid-structure dynamics make calibration especially difficult. Recent work on this platform [ 19 ] calibrated parameters by manually decomposing them into three stages, each with a tailored method (FFT matching, grid search, Bayesian optimization). Vid2Sid [ 26 ] demonstrated that iterative VLM feedback can replace hand-crafted metrics for system identification across multiple robotic platforms, but evaluated only up to 7 parameters with no downstream policy validation. Swim2Real scales VLM-guided calibration to 16 dimensions, introduces a backtracking line search that triples the accept rate to 42%, and validates the full pipeline through downstream RL training and real-world motor command transfer. Foundation Models in Robotics. Large language models (LLMs) and VLMs have been applied to task planning [ 1 ] , visuomotor control [ 2 ] , and robot co-design [ 25 , 24 ] . Eureka [ 16 ] uses LLMs to generate reward functions, and DrEureka [ 17 ] extends this to domain randomization distributions for sim-to-real transfer, automating the “train around the gap” approach rather than closing it. LLMs have also been used as iterative black-box optimizers [ 31 ] and to enhance Bayesian optimization surrogates [ 14 ] , but operate on scalar or text feedback in low-dimensional spaces. In contrast, Swim2Real uses a VLM for low-level parameter estimation , observing simulation-reality discrepancies in video to propose quantitative parameter adjustments in a 16-dimensional continuous space. III THE Swim2Real PIPELINE III-A

Read the complete text