Bridging Writing Manner Gap in Visual Instruction Tuning by Creating LLM-aligned Instructions

Dong Jing, Nanyi Fei, Zhiwu LuGaoling School of Artificial Intelligence, Renmin University of China {jingdong98, luzhiwu}@ruc.edu.cn · 2025

Read Bridging Writing Manner Gap in Visual Instruction Tuning by Creating LLM-aligned Instructions by Dong Jing, Nanyi Fei, Zhiwu LuGaoling School of Artificial Intelligence, Renmin University of China {jingdong98, luzhiwu}@ruc.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18320.txt.

Description

Models that see and read are trained on image instructions written by other models or by humans, in styles unlike the base model's own voice. Dong Jing, Nanyi Fei, and Zhiwu Lu argue that this style mismatch quietly damages the model. Forced to speak in a foreign manner, the base language model drifts from the strengths it was trained into.

Their fix lets the base model rewrite the instructions in its own manner, keeping the meaning intact. Human raters confirmed the rewritten instructions closed the style gap. Two well-known multimodal models trained on the aligned data resisted hallucination better. They also improved across fifteen vision and language benchmarks. The work was completed in late 2023.

Catalog record

CREATORDong Jing, Nanyi Fei, Zhiwu LuGaoling School of Artificial Intelligence, Renmin University of China {jingdong98, luzhiwu}@ruc.edu.cn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18320

Browse the full library · About the Tower of Babel