How Many Tools Should an LLM Agent See? A Chance-Corrected Answer
Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh, Joey Blackwell II · 2026
Read How Many Tools Should an LLM Agent See? A Chance-Corrected Answer by Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh, Joey Blackwell II (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24660.txt.
Description
Vyzantinos Repantis and others ask how many tools an agent should be shown. Too many muddles the choice, while too few can hide the right tool. No standard evaluation judged the shortlist length itself. They apply Bits-over-Random, a chance-corrected score, and turn it into a learning reward for picking depth per query. The learned policy nearly matches showing 50 tools on one benchmark while presenting only 7 on average. On hard queries where the right tool ranks low, it still finds 16.7 percent that a fixed shortlist misses.
Catalog record
| CREATOR | Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh, Joey Blackwell II |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37618 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24660