Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures

Shenran Wang, Timothy Tin-Long Tse, Jian Zhu, The University of British Columbia [email protected] , [email protected] , [email protected] · 2025

Read Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures by Shenran Wang, Timothy Tin-Long Tse, Jian Zhu, The University of British Columbia [email protected] , [email protected] , [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23006.txt.

Description

Shenran Wang, Timothy Tin-Long Tse, and Jian Zhu compare how different model designs learn from examples placed in the prompt. In-context learning lets a model do a new task from a few shown cases. Transformers are the usual design, but state-space and hybrid models are now strong rivals.

Across knowledge-based tasks, models with different internals scored alike yet worked differently inside. The team found function vectors, the carriers of the prompted task, mainly in attention and Mamba layers. They suggest Mamba2 may use a distinct route. The study warns that equal scores can hide unlike mechanisms.

Catalog record

CREATORShenran Wang, Timothy Tin-Long Tse, Jian Zhu, The University of British Columbia [email protected] , [email protected] , [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23006

Browse the full library · About the Tower of Babel