Retrieving Counterfactuals Improves Visual In-Context Learning

Guangzhi Xiong, Sanchit Sinha, Zhenghao He, Aidong Zhang · 2026

Read Retrieving Counterfactuals Improves Visual In-Context Learning by Guangzhi Xiong, Sanchit Sinha, Zhenghao He, Aidong Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16737.txt.

Description

Xiong, Sinha, He, and Zhang improve visual in-context learning by retrieving counterfactuals. Vision-language models often tangle fine visual traits with outcomes. In-context learning adapts them, but demo choice decides the gain. Similarity search picks correlated, non-causal examples and deepens spurious links. Their CIRCLES framework builds demo sets by composed image retrieval. Attribute-guided queries fetch counterfactual-style contrasts for the target. Models then reason about causes, not surface correlations. Gains were broad, largest for small models under scarce information.

Catalog record

CREATORGuangzhi Xiong, Sanchit Sinha, Zhenghao He, Aidong Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57749
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16737

Browse the full library · About the Tower of Babel