DisTaC : Conditioning Task Vectors via Distillation for Robust Model Merging

Kotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryotaro Shimizu, Hiroki Naganuma · 2025

Read DisTaC : Conditioning Task Vectors via Distillation for Robust Model Merging by Kotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryotaro Shimizu, Hiroki Naganuma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01148.txt.

Description

Kotaro Yoshida and four co-authors fortify, in 2025 work, the merging of trained models into one multi-task whole. Merges are cheap and flexy, yet past checks ran on sets that flatter the trick. On harder ground two flaws bite: task shift spans of unlike size, and source models that speak with low trust.

Their pre-step runs each source's lore through a teacher-student squeeze. The squeeze re-scales the shift span and lifts trust, while the task's core lore stays. Top merge tools then took in models with the bad traits — models they once failed on — and gained clear ground. Prep the parts, they show, and the merge holds.

Catalog record

CREATORKotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryotaro Shimizu, Hiroki Naganuma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01148

Browse the full library · About the Tower of Babel