What is smoltalk-chinese
smoltalk-chinese is an open-source synthetic dataset from OpenCSG designed specifically for Chinese large language models (LLMs). The dataset contains more than 700,000 synthetic data entries, covering a variety of task types such as information retrieval, reasoning, planning, editing, programming, mathematics, role-playing, data analysis, creative writing, consultation, and brainstorming. These diverse task designs are intended to enhance the model's versatility and adaptability, enabling better performance in different application scenarios. The dataset generation process strictly follows high standards, using advanced generative models and deduplication techniques to ensure data quality and diversity.
Main features of smoltalk-chinese
- Improve language model performance: The dataset is designed specifically for Chinese large language models (LLMs), supporting supervised fine-tuning (SFT) of models through high-quality synthetic data and improving model performance on a variety of tasks.
- Diverse task coverage: The dataset covers a variety of task types such as information retrieval, reasoning, planning, editing, programming, mathematics, role-playing, data analysis, creative writing, consultation, and brainstorming, enhancing the model's versatility and adaptability.
- High-quality data generation: Based on advanced generative models and deduplication techniques, it ensures data quality and diversity and avoids data duplication and redundancy.
- Support for multiple application scenarios: By simulating conversation styles in daily life and including data such as mathematics problems, the model can better adapt to real-world application scenarios.
Technical principles of smoltalk-chinese
- Data generation: Uses Magpie to synthesize raw data, combined with generative models such as deepseek-v2.5 and qwen2.5-72b-instruct, as well as the Distilabel library for data generation. The combination of these tools and models ensures the richness and diversity of the generated data.
- Data filtering: Based on the qwen2-7b-instruct model, the first instruction of the conversation data is scored for clarity and fluency, and only data with a score of 2 or above is retained to ensure data quality.
- Deduplication processing: Uses the gte-large-zh model to encode the first instruction of the conversation data, and performs deduplication based on embedding similarity (threshold set to 0.8) to ensure the uniqueness and diversity of the data.
- Data classification statistics: Classifies and counts the generated data to better understand the distribution and characteristics of the data.
Project address of smoltalk-chinese
- HuggingFace model library：https://huggingface.co/datasets/opencsg/smoltalk-chinese
Application scenarios of smoltalk-chinese
- Language model fine-tuning: The dataset is designed specifically for supervised fine-tuning (SFT) of Chinese large language models, supporting improved model performance on a variety of tasks through high-quality synthetic data.
- Diverse task training: The dataset covers a variety of task types such as information retrieval, reasoning, planning, editing, programming, mathematics, role-playing, data analysis, creative writing, consultation, and brainstorming, and can help models better understand and generate text in these fields.
- Dialogue system optimization: By simulating real user interaction scenarios, smoltalk-chinese provides rich training materials for dialogue systems, enabling them to better understand and generate natural language conversations.
- Improvement of mathematical reasoning ability: The dataset contains mathematical problem data from the Chinese version of Math23K, which helps enhance the model's capabilities in mathematical reasoning and problem solving.