Chinese-Vicuna: A Chinese Instruction-following Llama-based Model

Chenghao Fan, Zhenyi Lu, Jie Tian · 2025

Read Chinese-Vicuna: A Chinese Instruction-following Llama-based Model by Chenghao Fan, Zhenyi Lu, Jie Tian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12737.txt.

Introduction

Chenghao Fan and co-authors introduce Chinese-Vicuna, an open language model for Chinese instruction tasks, in this 2025 paper. The model fine-tunes Meta's LLaMA design with a low-rank adaptation method. It aims at users with modest hardware and small budgets.

The small model can run on a single consumer graphics card. Training mixed two open data sets and used a compact four-bit form. The model handles tasks like translation, code writing, and domain question answering. The project also ships tools for conversion, plain processor use, and multi-turn chat. It gives Chinese developers a cheap base for fields like health and law. The code and weights are open for others to adapt freely.

Catalog record

CREATORChenghao Fan, Zhenyi Lu, Jie Tian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chinese-Vicuna: A Chinese Instruction-following Llama-based Model Thanks: Authors are ordered alphabetically by the last name Chenghao Fan, Zhenyi Lu, Jie Tian Abstract Chinese-Vicuna is an open-source, resource-efficient language model designed to bridge the gap in Chinese instruction-following capabilities by fine-tuning Meta’s LLaMA architecture using Low-Rank Adaptation (LoRA). Targeting low-resource environments, it enables cost-effective deployment on consumer GPUs (e.g., RTX-2080Ti for 7B models) and supports domain-specific adaptation in fields like healthcare and law. By integrating hybrid datasets (BELLE and Guanaco) and 4-bit quantization (QLoRA), the model achieves competitive performance in tasks such as translation, code generation, and domain-specific Q&A. The project provides a comprehensive toolkit for model conversion, CPU inference, and multi-turn dialogue interfaces, emphasizing accessibility for researchers and developers. Evaluations indicate competitive performance across medical tasks, multi-turn dialogue coherence, and real-time legal updates. Chinese-Vicuna’s modular design, open-source ecosystem, and community-driven enhancements position it as a versatile foundation for Chinese LLM applications. 1 Introduction The advent of GPT-3.5 [ 2 ] marked a significant milestone in natural language processing, delivering impressive conversational capabilities that resonated across both academic and industrial sectors. This breakthrough spurred a surge of research focusing on scaling model sizes [ 11 , 10 , 3 ] , expanding datasets [ 11 ] , and refining instruction-based fine-tuning methods [ 4 ] . The open-source release of LLaMA [ 14 ] provided the community with a robust foundational model, catalyzing the development of various instruction-tuning techniques and frameworks. Researchers initially employed GPT-3.5 to generate responses for diverse queries, subsequently utilizing these outputs to fine-tune base models like Alpaca [ 13 ] . This iterative process has propelled the field forward, leading to more efficient and specialized large language models (LLMs). Despite the rapid advancements in large language models, the dominance of English-centric models like GPT-3.5 has highlighted a critical gap in efficient, localized solutions for languages such as Chinese [ 5 ] . The scarcity of robust, resource-efficient models for Chinese tasks has led to significant challenges in both accessibility and performance. While models like Chinese-LLaMA-Alpaca [ 5 ] attempt to address this by expanding vocabulary and pretraining, they still face the issue of high computational costs. In response, our Chinese-Vicuna offers a parameter-efficient alternative, prioritizing fine-tuning methods like LoRA [ 8 ] to adapt LLaMA for Chinese-specific tasks with minimal hardware requirements. Key innovations include: 1. Hardware Accessibility: Fine-tuning 7B/13B models on consumer GPUs (e.g., 30 hours on an RTX-2080Ti). 2. Vertical Domain Adaptation: Customizable workflows for medical diagnostics and legal analysis, leveraging structured datasets and LoRA’s modularity. 3. Deployment Flexibility: Support for 4-bit quantization [ 6 ] , CPU inference, and multi-GPU parallelism. This work highlights Chinese-Vicuna’s balance between performance and practicality, validated through domain-specific benchmarks and real-world use cases. 2 Related Work Researchers initially employed GPT-3.5 to generate responses for diverse queries, subsequently utilizing these outputs to fine-tune base models like Alpaca. This iterative process has propelled the field forward, leading to more efficient and specialized language models. Existing Chinese LLMs face two limitations: (1) Resource Intensity : Models like ChatGLM-6B [ 7 ] demand enterprise-grade GPUs, while Chinese-LLaMA-Alpaca’s expanded tokenizer [ 5 ] increases pretraining costs. (2) Domain Generalization : Projects such as BELLE [ 1 ] focus on broad instruction tuning but lack vertical specialization. Chinese-Vicuna builds on LoRA’s parameter efficiency, inspired by Guanaco’s 1 1 1 https://guanaco-model.github.io/ multilingual adaptability, but diverges by prioritizing Chinese-centric datasets and domain plugins. 3 Methodology To enhance the Chinese language capabilities of LLaMA-7B/13B, we first collected Chinese-related corpora, including general instruction data, dialogue data, and domain-specific data such as medical and legal texts. Further details can be found in the experimental section. To achieve training in low-resource scenarios, we combined quantization (8-bit or 4-bit) with LoRA, which significantly reduces the memory consumption during training. We implemented this approach using qlora from the ‘bitsandbytes’ 2 2 2 https://github.com/bitsandbytes-foundation/bitsandbytes library. We mixed the collected corpora and trained instruction-following models and dialogue models for both 7B and 13B. Specifically, we fine-tuned the 7B model with 8-bit quantization and the 13B model with 4-bit quantization, both using four 2080Ti GPUs. For Chinese downstream tasks, we employed the continued fine-tuning method. By constructing relevant prompts for each task and continuing the fine-tuning from intermediate checkpoints of the 7B or 13B models, we preserved the model’s training knowledge and further fine-tuned it on the downstream tasks. We found that continued fine-tuning yields better generalization compared to training from scratch on downstream tasks. It not only performs well on the current downstream tasks but also shows improved results on general tasks. Additionally, we identified two strategies that enhance the effectiveness of downstream task fine-tuning: 1. Single prompt: Using a single prompt yields better results than not using a prompt or using a mixture of multiple prompts. This approach minimizes the disruption to general instruction-following tasks during continued fine-tuning, while allowing the model to focus more on training for the current task. 2. Structuring the output: A simple structuring method, such as using ”1/2/3…” to format the output, enhances the logical coherence of the results. This chain-of-thought-style guidance helps the model exhibit stronger reasoning abilities during output generation. Our approach is built upon the LLaMA-7B and LLaMA-13B architectures, with modifications to better support the Chinese language through extended token embeddings. To enable efficient fine-tuning, we adopt the following strategies: 1. Parameter-Efficient Fine-Tuning: We utilize LoRA and QLoRA to insert trainable low-rank adaptation matrices into the frozen model layers, modifying only about 0.1% of the total parameters. Combined with 8-bit and 4-bit quantization, this significantly reduces VRAM requirements during training. 2. Domain Adaptation: Domain-specific datasets from the medical and legal fields are converted into instruction-response pairs for supervised fine-tuning. We follow a *continue training* paradigm, starting from a checkpoint that already possesses general conversational capabilities. This approach leads to more effective adaptation to downstream tasks. 3. Deployment: For deployment, we apply model quantization using GPTQ to enable efficient inference. The quantized models are compatible with CPU inference via ‘llama.cpp‘ 3 3 3 https://github.com/ggml-org/llama.cpp , and we provide a Gradio-based 4 4 4 https://github.com/gradio-app/gradio interactive interface for user-friendly access. 4 Experiments 4.1 Chinese Instruction-following Model 4.1.1 Dataset Overview We employ two primary datasets in our work: Belle [ 1 ] and Guanaco 5 5 5 https://guanaco-model.github.io/ . These datasets provide a diverse range of human instructions and responses, forming a crucial foundation for training a model capable of effectively following human instructions in Chinese. Guanaco The Guanaco dataset is designed to enhance multilingual capabilities and address a broad spectrum of linguistic tasks. It extends the 175 tasks from the Alpaca dataset by incorporating multilingual rewrites of seed tasks and introducing additional tasks specifically tailored for English grammar analysis, natural language understanding, cross-lingual self-awareness, and explicit content recognition. The dataset comprises a total of 534,530 samples, generated at a minimal cost of $6K. Belle The BELLE dataset is optimized for Chinese language understanding and instruction following. The fine-tuning process relies exclusively on data generated by ChatGPT, without integrating any external datasets. The dataset consists of approximately 500,000 Chinese instruction-response pairs produced by the BELLE project. Merged Dataset To construct a robust training corpus, we merge the Belle and Guanaco datasets, resulting in a dataset of nearly 700,000 samples.

Read the complete text