Mechanisms are Transferable: Data-Efficient Low-Resource Adaptation via Circuit-Targeted Supervised Fine-Tuning
Khumaisa Nur’aini, Ayu Purwarianti, Alham Fikri Aji, Derry Wijaya · 2026
Read Mechanisms are Transferable: Data-Efficient Low-Resource Adaptation via Circuit-Targeted Supervised Fine-Tuning by Khumaisa Nur’aini, Ayu Purwarianti, Alham Fikri Aji, Derry Wijaya (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08146.txt.
Description
Khumaisa Nur'aini and three co-authors ask whether learned skills move between languages inside a model. Their 2026 arXiv paper addresses low-resource language adaptation, where labeled data is scarce. Full fine-tuning there is unstable, and cross-lingual tuning can cause catastrophic forgetting.
The team proposes circuit-targeted supervised fine-tuning, called CT-SFT. It finds a sparse set of task-relevant attention heads in a proxy-language checkpoint. Scoring uses a label-balanced mean baseline and task-directional relevance, with no counterfactual step. Tuning only those circuits aims to transfer the mechanism, not just the words. The approach targets cheap adaptation that keeps prior knowledge intact.
Catalog record
| CREATOR | Khumaisa Nur’aini, Ayu Purwarianti, Alham Fikri Aji, Derry Wijaya |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08146