Classifier-Guided Captioning Across Modalities

Ariel Shaulov Tel Aviv University arielshaulov @mail.tau.ac.il Tal Shaharabany Tel Aviv University shaharabany @mail.tau.ac.il Eitan Shaar Bar-Ilan University [email protected] Gal Chechik Bar-Ilan University NVIDIA, Israel [email protected] Lior Wolf Tel Aviv University [email protected] · 2025

Read Classifier-Guided Captioning Across Modalities by Ariel Shaulov Tel Aviv University arielshaulov @mail.tau.ac.il Tal Shaharabany Tel Aviv University shaharabany @mail.tau.ac.il Eitan Shaar Bar-Ilan University [email protected] Gal Chechik Bar-Ilan University NVIDIA, Israel [email protected] Lior Wolf Tel Aviv University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03183.txt.

Description

Ariel Shaulov and four co-authors present classifier-guided captioning that works across modalities, in work from 2025. Most captioning systems are trained for one setting, such as images labeled by crowd workers, and adapt poorly to audio or video. Their method adapts a frozen captioning system at inference time, with no new training of the base model.

A text classifier, trained on data generated by GPT-4 with tailored prompts, steers captions toward the cues that matter in the new setting. For audio, that means naming sounds and their sources. Combined with an existing zero-shot audio captioning system, the method set a new best result, and tests spanned several models and modalities.

Catalog record

CREATORAriel Shaulov Tel Aviv University arielshaulov @mail.tau.ac.il Tal Shaharabany Tel Aviv University shaharabany @mail.tau.ac.il Eitan Shaar Bar-Ilan University [email protected] Gal Chechik Bar-Ilan University NVIDIA, Israel [email protected] Lior Wolf Tel Aviv University [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-03183

Browse the full library · About the Tower of Babel