Project Overview
OpenSeek is an open-source project initiated by the Beijing Academy of Artificial Intelligence (BAAI), with the goal of uniting the global open-source community to promote collaborative innovation at the algorithm, data, and system levels, and to develop the next-generation large language model that surpasses DeepSeek. Drawing on large-model initiatives such as Bigscience and OPT, the project is committed to building an independent open-source algorithmic innovation system.
After DeepSeek models were open-sourced, many algorithmic improvements and breakthroughs emerged in academia, but these innovations often lack complete code implementations, necessary computing resources, and high-quality data support. The OpenSeek project aims to work with the open-source community to explore mechanisms for building high-quality datasets, promote the open-sourcing of the entire large-model training process, build innovative training and inference code to support a variety of AI chips beyond Nvidia, and promote independent technological innovation and application development.
Project Goals
- Advanced data technology: solve the difficulty of obtaining high-quality data.
- Multi-AI device support: reduce dependence on specific chips and improve model generality and adaptability.
- Standardized LLM training baseline: promote independent algorithmic innovation and technology sharing through open-source collaboration.
Main Capabilities and Components
Standardized Baseline
openseek-baseline serves as the baseline for the PAZHOU algorithm competition and is also used to evaluate PRs in openseek. It is a standardized large-model training and evaluation pipeline, including a 100B dataset, training code, wandb, checkpoints, and evaluation results.
Datasets
The project has released the bilingual pretraining dataset CCI4.0-M2-V1 (released on May 6, 2025), with the goal of building a large-scale multilingual pretraining dataset of more than 10 trillion tokens, covering multiple languages and domains, and introducing data synthesis techniques such as chain-of-thought generation and instruction fine-tuning. In addition, OpenSeek-Pretrain-100B is a 100B data subset randomly sampled from CCI4.0-M2 v1, specifically for experimental training.
Models
The algorithm group has released the pretrained model OpenSeek-Small V1 (released on May 6, 2025, available for download).
Algorithm Group
Research focuses on three aspects of large language model training: data mixing, hyperparameter tuning, and reinforcement learning (RL). It systematically explores data composition strategies to balance cross-domain quality and diversity, studies the impact of hyperparameter configurations on training stability and convergence, and incorporates RL based optimization to further align model behavior with task-specific goals.
System Group
With support from the open-source community, flagscale aims to reproduce the distributed training system of DeepSeek V3 & R1, ensuring stable and high-performance end-to-end training.
Usage
Environment Preparation
Clone the repository and enter the directory, and install the FlagScale dependencies (Docker is recommended).
Prepare Data
Download the OpenSeek-Pretrain-100B dataset to a local directory and name it OpenSeek-Pretrain-100B. After building and activating the Python environment, you can also run a script to set up the project environment.
Run the Baseline
After completing environment installation and configuration, use a simple command to run the baseline:bash openseek/baseline/run_exp.sh start.
Verify Run
After running, a log directory OpenSeek-Small-v1-Baseline will be generated in the OpenSeek root directory; you can view the logs via vi OpenSeek-Small-v1-Baseline/logs/host_0_localhost.output . After about 1-2 minutes, execute grep "iteration.*consumed samples" OpenSeek-Small-v1-Baseline/logs/host_0_localhost.output; if the output resembles the example (including metrics such as iteration, consumed samples, throughput, etc.), it indicates that the program started successfully.
Use Cases
- Research and innovation in large language model algorithms
- Construction and experimentation of high-quality pretraining datasets
- Reproduction and optimization of distributed training systems
- Research on adaptation to multiple AI chips and model generality
- Open-source community collaboration and algorithm competitions (such as PAZHOU)
Limitations
Detailed usage restrictions, charging models, pricing, and other information are not yet provided. For contribution guidelines, Discord channels, and other ways to participate in the community, please refer to the project repository.
