In-Context Compositional Learning via Sparse Coding Transformer
Wei Chen, Jingxi Yu, Zichen Miao, Qiang Qiu · 2025
Read In-Context Compositional Learning via Sparse Coding Transformer by Wei Chen, Jingxi Yu, Zichen Miao, Qiang Qiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20194.txt.
Description
Chen, Yu, Miao, and Qiu rebuild transformer attention on sparse coding ideas. Their 2025 paper targets in-context compositional learning. Models must infer rules from examples given in context. Tasks combine basic parts through hidden compositional rules. Plain transformers lack the built-in structure for this.
Sparse coding writes data as few dictionary atoms combined. The team recasts attention as exactly that operation. An encoding dictionary turns inputs into sparse coefficients. Those coefficients mix a decoding dictionary into outputs. Sparsity forces the coefficients to expose real structure. Target answers get coefficients blended from context examples. Visual reasoning tests used the RAVEN benchmark family. S-RAVEN and RAVEN tasks measured compositional generalization. Their method held up where standard transformers failed. Unseen rule combinations were handled more gracefully. Learned structure transfers across tasks more cleanly here. Large pretrained models remain untested territory. The idea marries classic signal processing with attention.
Catalog record
| CREATOR | Wei Chen, Jingxi Yu, Zichen Miao, Qiang Qiu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20194