Mechanisms are Transferable: Data-Efficient Low-Resource Adaptation via Circuit-Targeted Supervised Fine-Tuning

Khumaisa Nur’aini, Ayu Purwarianti, Alham Fikri Aji, Derry Wijaya · 2026

Read Mechanisms are Transferable: Data-Efficient Low-Resource Adaptation via Circuit-Targeted Supervised Fine-Tuning by Khumaisa Nur’aini, Ayu Purwarianti, Alham Fikri Aji, Derry Wijaya (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08146.txt.

Description

Khumaisa Nur'aini and three co-authors ask whether learned skills move between languages inside a model. Their 2026 arXiv paper addresses low-resource language adaptation, where labeled data is scarce. Full fine-tuning there is unstable, and cross-lingual tuning can cause catastrophic forgetting.

The team proposes circuit-targeted supervised fine-tuning, called CT-SFT. It finds a sparse set of task-relevant attention heads in a proxy-language checkpoint. Scoring uses a label-balanced mean baseline and task-directional relevance, with no counterfactual step. Tuning only those circuits aims to transfer the mechanism, not just the words. The approach targets cheap adaptation that keeps prior knowledge intact.

Catalog record

CREATORKhumaisa Nur’aini, Ayu Purwarianti, Alham Fikri Aji, Derry Wijaya
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08146

Browse the full library · About the Tower of Babel