Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
Yuhui Zhang Yuchang Su Yiming Liu Xiaohan Wang James Burgess, Elaine Sui Chenyu Wang Josiah Aklilu Alejandro Lozano Anjiang Wei, Ludwig Schmidt Serena Yeung-Levy · 2025
Read Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation by Yuhui Zhang Yuchang Su Yiming Liu Xiaohan Wang James Burgess, Elaine Sui Chenyu Wang Josiah Aklilu Alejandro Lozano Anjiang Wei, Ludwig Schmidt Serena Yeung-Levy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03225.txt.
Description
Yuhui Zhang and co-authors present a 2025 framework that turns open visual questions into multiple-choice ones. Vision-language models need hard, fair tests, but open answers are hard to grade. Making good multiple-choice items by hand is slow and costly. Their system, AutoConverter, uses an agent pipeline to do the conversion. It builds wrong options that are tempting but not correct. Human-crafted quality is the bar it aims for.
In tests, the items it made were correct and genuinely challenging. Models scored on them much as they did on human-made sets. In some cases the machine items exposed bigger gaps between models. Cheaper test building could let benchmarks grow with the models. The work comes from a large team across several labs. It targets anyone who evaluates vision-language systems.
Catalog record
| CREATOR | Yuhui Zhang Yuchang Su Yiming Liu Xiaohan Wang James Burgess, Elaine Sui Chenyu Wang Josiah Aklilu Alejandro Lozano Anjiang Wei, Ludwig Schmidt Serena Yeung-Levy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03225