How Many Tools Should an LLM Agent See? A Chance-Corrected Answer

Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh, Joey Blackwell II · 2026

Read How Many Tools Should an LLM Agent See? A Chance-Corrected Answer by Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh, Joey Blackwell II (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24660.txt.

Description

Vyzantinos Repantis and others ask how many tools an agent should be shown. Too many muddles the choice, while too few can hide the right tool. No standard evaluation judged the shortlist length itself. They apply Bits-over-Random, a chance-corrected score, and turn it into a learning reward for picking depth per query. The learned policy nearly matches showing 50 tools on one benchmark while presenting only 7 on average. On hard queries where the right tool ranks low, it still finds 16.7 percent that a fixed shortlist misses.

Catalog record

CREATORVyzantinos Repantis, Ameya Gawde, Harshvardhan Singh, Joey Blackwell II
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37618
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24660

Browse the full library · About the Tower of Babel