Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success

George Bredis, Stanislav Dereka, Viacheslav Sinii · 2025

Read Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success by George Bredis, Stanislav Dereka, Viacheslav Sinii (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04280.txt.

Description

George Bredis, Stanislav Dereka, Viacheslav Sinii, and co-authors train vision-language models in cheap virtual worlds so the skills carry to real tasks. Models that see and read still struggle to turn what they see into a sound chain of actions. Prior training in simulators seldom proved that the skill would transfer.

Their method updates action words with a standard policy method while learning value only at the level of whole steps. Training in one simple simulator at a time led to gains on game control, spatial planning, and web tasks with real images. General image understanding stayed intact, which supports training in synthetic worlds.

Catalog record

CREATORGeorge Bredis, Stanislav Dereka, Viacheslav Sinii
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04280

Browse the full library · About the Tower of Babel