Towards Understanding What State Space Models Learn About Code

Jiali Wu, Abhinav Anand, Shweta Verma, Mira Mezini · 2026

Read Towards Understanding What State Space Models Learn About Code by Jiali Wu, Abhinav Anand, Shweta Verma, Mira Mezini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06774.txt.

Description

Jiali Wu, Abhinav Anand, Shweta Verma, and Mira Mezini open up state space models, the efficient rivals of transformers, to see what they actually learn about code. Earlier work showed these models can match or beat transformers on code tasks under fair training, but their inner workings stayed hidden. The paper gives the first systematic comparison of the two architectures on code understanding.

The analysis finds that state space models pick up syntactic and semantic structure better than transformers during pretraining, but forget some relations when fine-tuned on certain tasks. To diagnose that loss, the authors introduce a frequency-domain tool that shows a shift toward short-range dependencies during fine-tuning. Design changes guided by the diagnosis improved one code search benchmark by up to six points of mean reciprocal rank.

Catalog record

CREATORJiali Wu, Abhinav Anand, Shweta Verma, Mira Mezini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-06774

Browse the full library · About the Tower of Babel