Synergy: End-to-end Concept Model

Keli Zheng, Zerong Xie · 2025

Read Synergy: End-to-end Concept Model by Keli Zheng, Zerong Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12769.txt.

Introduction

Keli Zheng and Zerong Xie present a language model that learns its own chunks of text, in work from 2025. Their model, Synergy, works at the byte level and routes spans through layers of rising abstraction. Most models lean on a fixed tokenizer built before training. Synergy skips that step and finds its own units as it learns.

The model made fewer concept tokens than a standard byte tokenizer at like scores. Against a known open model at the same size and data, it held an edge. Mid layers did best with position codes removed, which hints at position-free concepts. The authors back tokenizer-free designs.

Catalog record

CREATORKeli Zheng, Zerong Xie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12769

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Synergy: End-to-end Concept Model Keli Zheng, Zerong Xie Abstract In this paper, we present Synergy , a language model that bridges different levels of abstraction in an end-to-end fashion through a learned routing mechanism. Focusing on low-level linguistic abstraction, we trained our model as a byte-level language model. Our model spontaneously learns to tokenize bytes, producing fewer concept tokens than Byte-level Byte Pair Encoder (BBPE) tokenizers whiling keeping the comparable performance. By comparing with Llama3 , we observed an advantage of Synergy under the same model scale and training dataset size. Further studies show that the middle part (the higher abstraction part) of our model performs better when positional encodings are removed, suggesting the emergence of position-independent concepts. These findings demonstrate the feasibility of tokenizer-free architectures, paving the way for more robust and flexible pipelines. 1 Introduction In the last few years, large language models (LLMs) have shown dazzling capabilities and enormous potential to solve a wide range of tasks. Many companies and research organizations have released their own large language models, such as ChatGPT, Llama, DeepSeek, etc., which have profoundly changed the way we work and live. Most of these products share a similar principle: a decoder-only-transformer-like model fed on a large number of tokens and trained with the predict-next-token task. Although researchers today have further developed sophisticated model structures and cleverer training approaches, the aforementioned principle still remains the core of most LLMs. However, as LCM-team et al. 2024 pointed out, such a core principle results in a drawback: it cannot efficiently process information at different levels of abstraction. Since the model is trained at the token level, it tends to “think” at the token level. However, many high-level abstract concepts are difficult to accurately describe in existing languages. It may prevent the model from performing better in scenarios where higher-level abstractions are critical, such as outlining a presentation, planning a tour, engineering a complicated program, etc. Many researchers believe that the ability to handle high-level abstract concepts can be spontaneously developed through training due to the multilayer structure of the transformer model. But a properly designed neural network structure can lead to a more efficient processing of high-level abstract concepts. Creatively, in the Large Concept Model (LCM) ( LCM-team et al. 2024 ) an autoencoder-based sentence embedding model is used to abstract token-level embeddings into sentence-level embeddings, and an encoder-only transformer model is fed with sentence-level embeddings and trained to predict next sentence embedding. Although their experiments have shown initial success, their solutions have an unavoidable disadvantage: The sentence embedding model is trained separately from the transformer model and have a quite different training target (in this case, as an autoencoder, to restore the original input), therefore the abstracted information may not be useful for the ultimate goal (in this case, to predict the next sentence embedding), leading to a low efficiency. To overcome this disadvantage, we propose an end-to-end model Synergy , where the encoding/decoding part and the processing part are trained together. The main challenge to train an end-to-end model is that the aforementioned two parts are working on token sequences with different lengths, which does not fit in a conventional decoder-only transformer. An encoder-decoder transformer is also not applicable due to the temporal dependency within/between these two sequences. Inspired by Mixture of Depths (MoD) ( Raposo et al. 2024 ) , we bridge those sequences using a router mechanism. Further, we configured those parts correspondingly to encourage a “division of labor” across different levels of abstraction, resulting in better efficiency. Tokens are also concepts at a low level of abstraction, except that they have already been symbolized in our language. Using statistic-based tokenizers like Byte-level Byte Pair Encoding (BBPE) tokenizers, we already achieved a quite efficient abstraction from characters/bytes to tokens/words. However, with the attempts to process multimodal data such as images, audio, video, etc. with LLMs, the ability to process data in low-level of abstractions becomes important. Considering the fact that low-level abstraction is relatively easier, and therefore consumes affordable resources, our experiments focused on low-level abstraction tasks. We compared our model Synergy with Llama3 , and observed an advantage under the same model scale and training dataset size. Further studies show that Synergy can generate fewer tokens than BBPE tokenizers. In addition, we observed an unexpected decrease of loss when removing positional encoding from the middle part, suggesting the emergence of position-independent concepts. In short, our contributions are: • We proposed an end-to-end model Synergy that bridges different levels of abstraction. • We conducted experiments to show the efficiency and feasibility of tokenizer-free Synergy as a byte-level language model. • We uncovered the emergence of position-independent concepts in the middle part of Synergy by removing positional encoding. 2 Methodology The main idea of Synergy is to reduce the number of tokens using a routing mechanism, then process the resulting shorter sequence using a decoder-only transformer. Like a normal decoder-only transformer, the whole model consumes a sequence of tokens as input, produces a corresponding sequence of tokens as output, and is trained by predict-next-token tasks. As it shows in fig. 1(a) , the model is split into three parts — encoder, middle, and decoder, which are all decoder-only transformers. The output from the encoder is passed to the decoder. For a portion of tokens, the encoder output also passes through the middle part and then adds onto the decoder input. A router is responsible to determine which tokens can go through the middle part. As it shows in figs. 1(b) and 1(c) , the routing mechanism is similar to that in MoD ( Raposo et al. 2024 ) . A weight factor w_{i} is computed from the encoder output x_{i} through a simple linear layer for each token (indexed by i ). A portion of tokens are picked to pass through the middle part via a top-k operation on the weight factor. The picking result is represented as a mask m_{i} . For the picked tokens, the middle output \text{Middle}(x_{i}) is multiplied by a gating factor \sigma_{i} before adding to the decoder input. The routing mechanism can be formally described as the following formulas: \displaystyle w_{i} \displaystyle=\text{Router}(x_{i}) (1) \displaystyle m_{i} \displaystyle=\text{topkmask}(w_{i},k) (2) \displaystyle\sigma_{i} \displaystyle=\text{sigmoid}(w_{i}) (3) \displaystyle y_{i} \displaystyle=x_{i}+m_{i}\ \sigma_{i}\ \text{Middle}(x_{i}) (4) Where, • x_{i} — The encoder output of the i -th token. • w_{i} — The weight factor of the i -th token computed by the router. • k — A hyperparameter that controls the number of tokens that can pass through the middle part. • m_{i} — The mask that represents whether the i -th token can pass through the middle part. • \sigma_{i} — The gating factor of the i -th token computed from w_{i} . • y_{i} — The decoder input of the i -th token. (a) The overview (b) Routing (active) (c) Routing (inactive) Figure 1: The architecture of Synergy Here we can consider the weight factor w_{i} as an importance indicator of each token. While a token is important, it is reasonable to allocate more computation to process it, so it activates the middle part. As a result, the middle part processes fewer tokens than the encoder/decoder part, achieving our goal of token number compressing. We assume

Read the complete text