Test-Time Reinforcement Learning for GUI Grounding via Region Consistency

Yong Du 1,2,∗ , Yuchen Yan 1,∗ , Fei Tang 1 , Zhengxi Lu 1 , Chang Zong 3 , Weiming Lu 1 , Shengpei Jiang 4 Yongliang Shen 1,† 1 Zhejiang University 2 Central South University 3 Zhejiang University of Science and Technology 4 SF Technology [email protected] {yanyuchen,syl}@zju.edu.cn GitHub: https · 2025

Read Test-Time Reinforcement Learning for GUI Grounding via Region Consistency by Yong Du 1,2,∗ , Yuchen Yan 1,∗ , Fei Tang 1 , Zhengxi Lu 1 , Chang Zong 3 , Weiming Lu 1 , Shengpei Jiang 4 Yongliang Shen 1,† 1 Zhejiang University 2 Central South University 3 Zhejiang University of Science and Technology 4 SF Technology [email protected] {yanyuchen,syl}@zju.edu.cn GitHub: https (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05615.txt.

Description

Yong Du and co-authors let a screen-reading model grade its own homework. GUI grounding means turning an instruction like tap the settings gear into exact screen coordinates, a core skill for phone and computer agents. Training it normally needs costly pixel-level labels. The team noticed that when a model guesses the same element many times, the guesses cluster where the model is most sure.

Their test-time method builds a voting grid from those samples and reads off the consensus region, gaining two to three points with no training at all. A second step turns agreement into a reward and lets the model polish itself live on unlabeled screens. With only 1,272 unlabeled examples, that lifted accuracy by three to six points across model types on standard benchmarks.

Catalog record

CREATORYong Du 1,2,∗ , Yuchen Yan 1,∗ , Fei Tang 1 , Zhengxi Lu 1 , Chang Zong 3 , Weiming Lu 1 , Shengpei Jiang 4 Yongliang Shen 1,† 1 Zhejiang University 2 Central South University 3 Zhejiang University of Science and Technology 4 SF Technology [email protected] {yanyuchen,syl}@zju.edu.cn GitHub: https
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05615

Browse the full library · About the Tower of Babel