Long Exposure : Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity

Tuowei Wang 1 2 3 4 , Kun Li 1 4 5 , Zixu Hao 1 2 , Donglin Bai 1 , Ju Ren 2 5 , Yaoxue Zhang 2 , Ting Cao 1 , Mao Yang 1 · 2025

Read Long Exposure : Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity by Tuowei Wang 1 2 3 4 , Kun Li 1 4 5 , Zixu Hao 1 2 , Donglin Bai 1 , Ju Ren 2 5 , Yaoxue Zhang 2 , Ting Cao 1 , Mao Yang 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-15964.txt.

Description

Tuowei Wang leads seven co-authors in speeding up parameter-efficient fine-tuning of large language models. Such tuning adapts a pre-trained model by changing few added parameters. It still costs much time and money in practice. The team names a neglected pattern in that process. They call it Shadowy Sparsity, a subtle form of sparsity that is specific to fine-tuning.

Their system, Long Exposure, attacks the pattern in three parts. An exposer senses over a longer range to catch more hidden sparsity. A sequence-oriented predictor handles long inputs and shifting parameters. A dynamic-aware operator structures the sparse math for cleaner memory access. Across tests, the system beats leading prior methods. End-to-end fine-tuning speedups reach about two and a half times at best. The work targets cheaper adaptation of large models to new tasks.

Catalog record

CREATORTuowei Wang 1 2 3 4 , Kun Li 1 4 5 , Zixu Hao 1 2 , Donglin Bai 1 , Ju Ren 2 5 , Yaoxue Zhang 2 , Ting Cao 1 , Mao Yang 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-15964

Browse the full library · About the Tower of Babel