ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents

Xiaoce Wang, Guibin Zhang, Junzhe Li, Jinzhe Tu, Chun Li, Ming Li · 2026

Read ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents by Xiaoce Wang, Guibin Zhang, Junzhe Li, Jinzhe Tu, Chun Li, Ming Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02548.txt.

Description

Xiaoce Wang and 5 co-authors replace one-shot coordinate prediction with stepwise tool use for screen agents, in a 2026 arXiv paper. Agents that click by predicting coordinates generalize poorly across screen sizes and shapes. Coordinate-free options need more training data than exists.

ToolTok models screen operations as a sequence of progressive tool calls, in line with human interaction habits. Each tool is a learnable token embedding. An anchoring mechanism grounds every tool in related concepts, giving a helpful head start so embeddings can be learned from little supervision. A three-task curriculum, from token definitions through text-guided selection to simplified visual pathfinding, teaches a pretrained model the tool meanings step by step. The four-billion-parameter system beat models of like size and rivaled one nearly sixty times larger, using under one percent of rivals' training data.

Catalog record

CREATORXiaoce Wang, Guibin Zhang, Junzhe Li, Jinzhe Tu, Chun Li, Ming Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02548

Browse the full library · About the Tower of Babel