Adaptive Dropout for Pruning Conformers
Yotaro Kubo, Xingyu Cai, Michiel Bacchiani · 2024
Read Adaptive Dropout for Pruning Conformers by Yotaro Kubo, Xingyu Cai, Michiel Bacchiani (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04836.txt.
Description
Yotaro Kubo, Xingyu Cai, and Michiel Bacchiani teach speech recognition networks to prune themselves while they train, in this 2024 paper. Their target is the Conformer, the workhorse architecture of modern speech recognition, whose size taxes deployment. Separate pruning after training usually costs accuracy.
Their method inserts adaptive dropout layers that learn a retention probability for every unit. Units the model keeps switching off reveal themselves as prunable. The Gumbel-Softmax trick keeps the whole thing trainable by back-propagation. Layers were placed at three points in each Conformer block: the feed-forward hidden layer, the attention queries and values, and the local convolution inputs. On LibriSpeech, the pruned model did the rare double: 54 percent fewer parameters and word error rates about one point better.
Catalog record
| CREATOR | Yotaro Kubo, Xingyu Cai, Michiel Bacchiani |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04836