Towards Understanding Self-Pretraining for Sequence Classification

Omar Coser, Loredana Zollo, Paolo Soda, Antonio Orvieto · 2026

Read Towards Understanding Self-Pretraining for Sequence Classification by Omar Coser, Loredana Zollo, Paolo Soda, Antonio Orvieto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21070.txt.

Description

Omar Coser and three colleagues study self-pretraining for sequence sorting. Amos and team showed in 2024 that transformer accuracy in sequence sorting can be raised by first pretraining with a masked token guessing objective without outside data or extra data. While their goal was strong performance on the Long-Range Arena, their pipeline raises fundamental questions. How does SPT drive tuning to better solutions, and why can standard labeled training fail in those models?

Catalog record

CREATOROmar Coser, Loredana Zollo, Paolo Soda, Antonio Orvieto
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64115
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21070

Browse the full library · About the Tower of Babel