ScreenExplorer: Training a Vision–Language Model for Diverse Exploration in Open GUI World

Runliang Niu, Jinglong Ji, Yi Chang, Qi Wang · 2025

Read ScreenExplorer: Training a Vision–Language Model for Diverse Exploration in Open GUI World by Runliang Niu, Jinglong Ji, Yi Chang, Qi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19095.txt.

Description

Runliang Niu, Jinglong Ji, Yi Chang, and Qi Wang train a vision-language model to explore computer interfaces on its own, in a paper from 2025. Screen agents usually memorize fixed task data and fail in new apps. Learning by exploring live environments could break that dependence on hand-made datasets.

Training ran inside real, changing graphical interfaces with a group-based reinforcement method. A world-model curiosity reward pushed the agent through the awkward cold-start phase when nothing works yet. Distilling its own experience streams further sharpened exploration. The trained model adapted to new environments better and kept exploring longer than static counterparts.

Catalog record

CREATORRunliang Niu, Jinglong Ji, Yi Chang, Qi Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19095

Browse the full library · About the Tower of Babel