A Perplexity and Menger Curvature-Based Approach for Similarity Evaluation of Large Language Models
Yuantao Zhang, Zhankui Yang · 2025
Read A Perplexity and Menger Curvature-Based Approach for Similarity Evaluation of Large Language Models by Yuantao Zhang, Zhankui Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04216.txt.
Introduction
Zhang and Yang propose a new way to judge how similar two large language models are to each other. The concern is copying: a slightly altered model could be passed off as a brand new one. Their 2025 paper builds a score from how each model rates text, using curves tied to a geometry measure called Menger curvature.
Tests show the method beats standard baselines and works across varied models and domains. In simulations, it also caught cases of model copying. The authors offer the score as a guard for ownership rights and honest claims. Code for the method is shared in a public repository, they note. The score is meant for owners, buyers, and auditors of models.
Catalog record
| CREATOR | Yuantao Zhang, Zhankui Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-04216
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Perplexity and Menger Curvature-Based Approach for Similarity Evaluation of Large Language Models Yuantao Zhang, Zhankui Yang Abstract The rise of Large Language Models (LLMs) has brought about concerns regarding copyright infringement and unethical practices in data and model usage. For instance, slight modifications to existing LLMs may be used to falsely claim the development of new models, leading to issues of model copying and violations of ownership rights. This paper addresses these challenges by introducing a novel metric for quantifying LLM similarity, which leverages perplexity curves and differences in Menger curvature. Comprehensive experiments validate the performance of our methodology, demonstrating its superiority over baseline methods and its ability to generalize across diverse models and domains. Furthermore, we highlight the capability of our approach in detecting model replication through simulations, emphasizing its potential to preserve the originality and integrity of LLMs. Code is available at https://github.com/zyttt-coder/LLM_similarity . 1 Introduction In the past year, the rapid development of Large Language Models (LLMs) and their wide application have become a hot spot in different domains. Although LLMs provide a more convenient way to acquire knowledge and solve problems, they also bring about some issues. Companies and organizations have begun to exploit LLMs for profit by engaging in unethical practices such as directly copying model structures and codes, and violating open-source licenses. For instance, the Yi-34B model developed by Chinese 01-ai company uses exactly Llama’s architecture except for two tensors renamed 1 1 1 hf.co/01-ai/Yi-34B/discussions/11 . Additionally, there are cases where proprietary LLMs are rebranded with slight modifications, such as adding noise, and falsely represented as original creations. Recent studies have revealed that the Llama3-V project code from Stanford team is a reformulation of the MiniCPM-Llama3-V2.5 Xu et al. (2024a) ; Yu et al. (2024) , and the behavior of the Llama3-V model is very similar to a noised version of the MiniCPM-Llama3-V2.5 checkpoint. Besides, methods can be used to distill the knowledge of a LLM in specific areas into other LLMs Xu et al. (2024b) . Without explicit clarifications, such actions may potentially breach the model’s user policies. While distilled models and the original LLM have similarity in the distilled areas, their performance could diverge in other domains, making detection more difficult. The practices mentioned above significantly undermine companies’ exclusive rights to their own products, highlighting the need for effective approaches to measure the similarity between LLMs and uncover these activities. Figure 1: Perplexity curve of gpt2 on the text "Turkish (Türkçe) is a language officially spoken in Turkey and Northern Cyprus." Each point represents the perplexity of the sequence from word index 0 to its respective word index. Assessing the similarity of LLMs is a complex task, particularly when models are not fully open-source, a scenario that remains prevalent. In many cases, models may have publicly available parameter weights, yet their training datasets and processes are undisclosed. Although parameter space comparisons are feasible in such cases, estimating domain-specific model differences based solely on parameter comparisons remains a challenge. For LLMs with undisclosed parameters, existing methods evaluating similarity are underexplored. An intuitive approach involves comparing model outputs given identical prompts. However, even when outputs differ, models may still be similar due to underlying probability distributions. Another approach is to use evaluation benchmarks and metrics such as accuracy and BERTScore Zhang et al. (2019) to assess performance similarity between models, but this method also lacks robustness, as shown in Table 1 . If additional information, such as next-token probabilities, is available (e.g. the GPT-4 model Achiam et al. (2023) ), alternative approaches can be applied. Since most auto-regressive LLMs are trained to maximize the likelihood of each token based on preceding tokens Floridi and Chiriatti (2020) , comparing the closeness of next-token distributions could provide insights into model similarity. However, the computational costs and applicability of such methods remain insufficiently studied. Models Llama(7b) Pythia(6.9b) Pythia(160m) neo(125m) PIQA 0.798 0.76 0.618 0.631 Table 1: Zero-shot performance on the PIQA benchmark Bisk et al. (2020) . The scores of Llama7B and Pythia-6.9b on the PIQA benchmark are very close, as well as the scores of Pythia-160m and neo-125m, yet they are distinct LLMs. Therefore, this work focuses on the similarity comparison of LLMs and its application across different domains, as well as its potential use in detecting model replication. We propose a novel metric to quantify LLM similarity, utilizing the perplexity curve (shown in Figure 1 ) and the difference of Menger curvature Léger (1999) to represent the degree of similarity between LLMs. To sum up, we make the following contributions in this work: 1. We address the challenge of distinguishing a model coming from existing models by some simple methods, such as modifying model parameters or reformatting code. To this end, we develop a quantitative approach to measure LLM similarity. 2. We validate the feasibility of our proposed metric through preliminary experiments and demonstrate that it outperforms several baseline methods. Furthermore, we expand our approach to include a broader range of LLMs and evaluation datasets across various domains. 3. We simulate a model-copying scenario by introducing noise into model parameters and establish thresholds for identifying copied LLMs, demonstrating the practical applicability of our method in real-world contexts. 2 Related work 2.1 Perplexity Perplexity is a metric to measure the degree uncertainty in predicting the next token in a sequence based on preceding tokens. It is calculated using the negative average log-likelihood of texts under a language model Brown et al. (1992) . The formula for perplexity is defined as: \text{PPL}(x)=exp[-\frac{1}{t}\sum_{i=1}^{t}\log p(x_{i}|x_{<i})] where x is a sequence of t tokens, as described by Alon and Kamfonas (2023) . Here, p(x_{i}|x_{<i}) denotes the surprise of prediction, referring to the next-token probability discussed in Section 1 . Perplexity is widely used to detect LLM-generated texts Tang et al. (2024) . Research indicates that language models often concentrate on typical patterns in their training data, resulting in low perplexity scores for LLM-generated texts. In contrast, human-generated texts tend to exhibit higher perplexity values due to their varied styles of expression. Based on this observation, Mitchell et al. (2023) developed DetectGPT, employing probability curvature to detect machine-generated texts. DetectGPT’s team finds that the change of log perplexity when applying perturbation to a text fragment is different for human-written texts and AI-generated texts. Building on the foundation of DetectGPT, Xu and Sheng (2024) designed AIGCode detector, which examines the perplexity change of code pieces after perturbation to discover AI-generated codes. While perplexity displays broad utility, research on its variation within a single sentence remains limited. Our method studies perplexity changes in a different manner and within a distinct application context. 2.2 Pairwise Comparison of LLMs While the evaluation of a single LLM is well-established Liang et al. (2022) ; Chang et al. (2024) , recent research has begun to emphasize pairwise evaluations of LLMs. Motivated by the fact that comparing two options rather than scoring each one independently is more intuitive from a human perspective, Liusie et al. (2023) examines comparative assessment across multiple dimensions, concluding that it offers a simple, general and effective approach for NLG (Natural Language Generation) assessment. Kahng et al. (2024) introduces LLM Comparator, an innovative visual analytics tool for interactively analyzing results from automatic side-by-side