Adaptive Dropout for Pruning Conformers

Yotaro Kubo, Xingyu Cai, Michiel Bacchiani · 2024

Read Adaptive Dropout for Pruning Conformers by Yotaro Kubo, Xingyu Cai, Michiel Bacchiani (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04836.txt.

Description

Yotaro Kubo, Xingyu Cai, and Michiel Bacchiani teach speech recognition networks to prune themselves while they train, in this 2024 paper. Their target is the Conformer, the workhorse architecture of modern speech recognition, whose size taxes deployment. Separate pruning after training usually costs accuracy.

Their method inserts adaptive dropout layers that learn a retention probability for every unit. Units the model keeps switching off reveal themselves as prunable. The Gumbel-Softmax trick keeps the whole thing trainable by back-propagation. Layers were placed at three points in each Conformer block: the feed-forward hidden layer, the attention queries and values, and the local convolution inputs. On LibriSpeech, the pruned model did the rare double: 54 percent fewer parameters and word error rates about one point better.

Catalog record

CREATORYotaro Kubo, Xingyu Cai, Michiel Bacchiani
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04836

Browse the full library · About the Tower of Babel