Evolutionary Prompt Optimization Discovers Emergent Multimodal Reasoning Strategies in Vision-Language Models

Sid Bharthulwar, John Rho, Katrina Brown · 2025

Read Evolutionary Prompt Optimization Discovers Emergent Multimodal Reasoning Strategies in Vision-Language Models by Sid Bharthulwar, John Rho, Katrina Brown (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23503.txt.

Description

Sid Bharthulwar, John Rho, and Katrina Brown present a 2025 paper on evolving better prompts for vision language models. A prompt can be tuned like a trait under selection, with the best variants surviving each round. That style of search may find tricks no human would write by hand.

Their framework runs prompt variants through rounds of testing on visual tasks, keeping the winners. Across generations, the model struck on fresh tactics on its own. One tactic was to call a code tool to crop or recolor an image before answering. The authors show that naming this tool use in the prompt lifts scores a lot.

Catalog record

CREATORSid Bharthulwar, John Rho, Katrina Brown
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23503

Browse the full library · About the Tower of Babel