MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models

Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, Zheng Zhang · 2025

Read MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models by Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, Zheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11513.txt.

Description

Zhen Zhang and co-authors fine-tune large models on many tasks without backpropagation in a 2025 paper. Zeroth-order methods save the memory that gradients demand. Their gradient estimates are noisy, holding them back. Multi-task use was unexplored and amplifies that noise.

Their framework attacks the problem at the parameter level. A weight importance metric finds the parameters that matter. A mask then updates only those critical weights. Shrinking the update space cuts variance directly. Task conflicts shrink along with the space. Experiments show state-of-the-art results for the setting. Even first-order multi-task methods were surpassed. Shared knowledge across tasks still transfers under the mask. Resource-constrained labs can fine-tune bigger models. Memory walls, not compute, gate much fine-tuning today. The mask idea may generalize to other optimizers.

Catalog record

CREATORZhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, Zheng Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11513

Browse the full library · About the Tower of Babel