Llamion Technical Report

Kisu Yang, Yoonna Jang, Hyeonseok Moon, Hwanseok Jang, Taewoo Lee, Hyungjin Lee, Jeseung Lee, Juhyoung Park, Heuiseok Lim, [0.3em] VAIV Company · 2026

Read Llamion Technical Report by Kisu Yang, Yoonna Jang, Hyeonseok Moon, Hwanseok Jang, Taewoo Lee, Hyungjin Lee, Jeseung Lee, Juhyoung Park, Heuiseok Lim, [0.3em] VAIV Company (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25676.txt.

Introduction

Yang and colleagues release Llamion, open-weight language models with 14B parameters built by converting Orion-14B into the standard Llama-family architecture. Their KEPT recipe maps unchanged modules, starts LayerNorm-to-RMSNorm conversion with a provably optimal training-free setup, and distills the converted model against the frozen source teacher. Llamion recovers Orion's behavior on H6, MT-Bench and KoMMLU, using about 123M tokens on one A100 in four days. Its base version reaches 66.87 percent on KoMMLU, beating the next-best open Korean entry by over seven points.

Catalog record

CREATORKisu Yang, Yoonna Jang, Hyeonseok Moon, Hwanseok Jang, Taewoo Lee, Hyungjin Lee, Jeseung Lee, Juhyoung Park, Heuiseok Lim, [0.3em] VAIV Company
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39668
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25676

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Llamion Technical Report Kisu Yang, Yoonna Jang, Hyeonseok Moon, Hwanseok Jang, Taewoo Lee, Hyungjin Lee, Jeseung Lee, Juhyoung Park, Heuiseok Lim, [0.3em] VAIV Company Abstract We release Llamion , a family of 14B-parameter open-weight language models obtained by transforming Orion-14B into the standardized Llama-family architecture. The transformation is performed by Efficient Knowledge Preservation for Transformation ( KEPT ), a recipe that combines (i) Normal Parameter Mapping (NPM) for unchanged modules, (ii) Optimized Parameter Mapping (OPM), a training-free LayerNorm-to-RMSNorm initialization we prove optimal under the near-zero-mean activation regime induced by weight decay, and (iii) Cross-architecture Knowledge Distillation (XKD), an equal-size frozen-teacher distillation that aligns the converted model’s outputs with the source model’s on any reasonable input distribution. Llamion recovers Orion’s behaviour on H6, MT-Bench, and KoMMLU with only ∼ 123 {\sim}123 M tokens on a single A100 in four days; Llamion-Base reaches 66.87% on KoMMLU, exceeding the next-best entry of the Open Ko LLM Leaderboard by > 7.0 {>}7.0 absolute points at submission time. Capabilities entirely absent from the transfer corpus — Python programming and 200K-token context handling — survive the architectural transition intact. We release three checkpoints (Base, Chat, LongChat) that load with trust_remote_code=False in the Hugging Face Transformers library. 1 Introduction The Transformer ( 34 ) has become the universal template for LLMs, yet within that template the open-source community continues to make divergent low-level choices: LayerNorm ( 3 ) vs. RMSNorm ( 44 ) , multi-head vs. grouped-query attention ( 1 ) , absolute/ALiBi/RoPE ( 32 ) encodings, GeLU vs. SwiGLU activations. Although the field has converged on a dominant family-level template — a decoder-only stack with RoPE, RMSNorm, and SwiGLU, shared by Llama-2/3 ( 33 ; 10 ) , Qwen2.5 ( 30 ) , DeepSeek-V3 ( 9 ) , and Gemma-2 ( 12 ) — earlier or third-party checkpoints (e.g. those built on LayerNorm or non-standard attention) remain valuable but increasingly out of step with this template. Practitioners therefore regularly need to convert a competent pretrained model from one set of choices to another, for compatibility with downstream tooling, deployment on a particular runtime, or to graft newer architectural ideas onto an existing checkpoint. Architectural transformation is chronically fraught with knowledge loss ( 23 ; 28 ) . Even mechanical conversions such as multi-head \to grouped-query attention ( 1 ) alter the network’s effective computational graph enough that previously learnt knowledge is partially destroyed. The conventional remedy is uptraining : continued training on additional data to let the modified network re-discover the lost capabilities. Uptraining works but at considerable cost. The GQA paper ( 1 ) reports recovering the modified model with {\sim}5\% of the original pretraining compute, and the new corpus must cover everything the model previously knew, on pain of catastrophic forgetting ( 14 ; 19 ) . For modern LLMs whose pretraining corpora are not public, this curation is often impossible. This report introduces a different remedy. We present KEPT 1 1 1 The acronym is arranged for clarity despite a slight word-order rearrangement of Efficient Knowledge Preservation for Transformation . , which rather than retrain the modified network on a new corpus, aligns its outputs with those of the original network on any reasonable input distribution. The frozen original network plays the role of an equal-size teacher; the modified network, initialized by a parameter-level mapping from the teacher, plays the role of a student that need only learn to reproduce the teacher’s hidden states or logits. Because the alignment target is the teacher’s behaviour rather than the original training data, KEPT does not require access to the pretraining corpus, does not bottleneck on curation, and produces a student that can be drop-in replaced for the teacher at deployment time. KEPT comprises Normal Parameter Mapping (NPM) for modules whose computation is unchanged, Optimized Parameter Mapping (OPM) for modules whose computation has changed but admits a principled initialization, and Cross-architecture Knowledge Distillation (XKD) for the residual alignment. Our central theoretical result is a closed-form solution for OPM: we prove that the optimal RMSNorm weights replacing a LayerNorm under L2 output alignment equal the LayerNorm weights themselves in the near-zero-mean regime induced by weight decay (Appendix A ). This is what makes OPM training-free and distinguishes KEPT from naive parameter copying or generic distillation: the dominant residual gap in this class of conversions is solved analytically rather than empirically. We instantiate KEPT through a deliberately worst-case conversion: Orion-14B ( 6 ) , a model competent in East Asian languages but built on a third-party architecture (LayerNorm, ad-hoc attention layout, third-party loader), into the standardized Llama-family template ( 33 ) . The resulting model Llamion inherits Orion’s Korean strength while becoming directly usable in mainstream tooling. The Orion \to Llama pair maximizes architectural divergence, which is the setting in which KEPT most needs to be tested; the method itself requires only that source and target share a stacked-block, residual-stream structure (§ 4.1 ). Empirically, KEPT recovers the source model’s behaviour using {\sim}123 M training tokens on a single A100 GPU over four days, far below uptraining budgets. On H6, Llamion’s average is within 0.3 points of Orion’s; on MT-Bench, the multi-turn average matches Orion’s within 0.1 points; on KoMMLU, Llamion-Base reaches 66.87%, exceeding the next-best entry of the Open Ko LLM Leaderboard by {>}7 absolute points at submission time. Crucially, capabilities entirely absent from the transfer corpus — Python programming, 200K-token context handling — survive the architectural transition intact (§ 7 ). These zero-shot transfer effects are, to our knowledge, the most direct evidence that KEPT preserves what the teacher knew rather than merely what the transfer corpus contained. Released artifacts. We release three Llamion checkpoints: Base , Chat , and LongChat — on the Hugging Face Hub as the vaiv/GeM2-Llamion 2 2 2 https://huggingface.co/collections/vaiv/gem2-llamion collection. All three load with trust_remote_code=False in the standard transformers library. 2 Background and Related Work Uptraining and architectural transformation. Uptraining — continued training of an architecturally modified network ( 23 ; 28 ) — is the dominant remedy when an LLM’s architecture is altered. The original GQA recipe ( 1 ) reports recovery at {\sim}5\% of pretraining compute; Linearizing LLMs ( 28 ) and Mamba-in-Llama ( 35 ) apply the same principle to more aggressive transitions, the latter distilling a Llama-architecture Transformer into a Mamba–attention hybrid by reusing attention weights. A complementary line modifies the architecture while preserving the parameter count: Sparse Upcycling ( 23 ) expands dense MLPs into MoE; SOLAR ( 20 ) performs depth up-scaling; LLaMA-Pro ( 39 ) inserts identity-initialized blocks for block expansion. EEVE ( 21 ) performs efficient vocabulary expansion for Korean, illustrating the same general pattern of cheap modification via principled initialization plus light continued training . Uptraining’s chief limitations are twofold: it requires data covering all knowledge to be preserved (impossible when the pretraining corpus is private), and the compute, even when the data is curated, can reach a non-trivial fraction of pretraining. Knowledge distillation. Classical KD trains a small student to mimic a larger teacher’s outputs ( 5 ; 2 ; 17 ) , classified as offline (frozen teacher) ( 17 ) , online ( 46 ) , or self-distillation ( 42 ; 36 ; 40 ) ; see 41 ; 15

Read the complete text