Bridging Writing Manner Gap in Visual Instruction Tuning by Creating LLM-aligned Instructions
Dong Jing, Nanyi Fei, Zhiwu LuGaoling School of Artificial Intelligence, Renmin University of China {jingdong98, luzhiwu}@ruc.edu.cn · 2025
Read Bridging Writing Manner Gap in Visual Instruction Tuning by Creating LLM-aligned Instructions by Dong Jing, Nanyi Fei, Zhiwu LuGaoling School of Artificial Intelligence, Renmin University of China {jingdong98, luzhiwu}@ruc.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18320.txt.
Description
Models that see and read are trained on image instructions written by other models or by humans, in styles unlike the base model's own voice. Dong Jing, Nanyi Fei, and Zhiwu Lu argue that this style mismatch quietly damages the model. Forced to speak in a foreign manner, the base language model drifts from the strengths it was trained into.
Their fix lets the base model rewrite the instructions in its own manner, keeping the meaning intact. Human raters confirmed the rewritten instructions closed the style gap. Two well-known multimodal models trained on the aligned data resisted hallucination better. They also improved across fifteen vision and language benchmarks. The work was completed in late 2023.
Catalog record
| CREATOR | Dong Jing, Nanyi Fei, Zhiwu LuGaoling School of Artificial Intelligence, Renmin University of China {jingdong98, luzhiwu}@ruc.edu.cn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18320