ForgeTrain - The open-source large model pre-training framework from ModelBest in collaboration with Tsinghua and others

What is ForgeTrain

ForgeTrain is the world's first production-grade large model pre-training framework released as open source by ModelBest in collaboration with Tsinghua University and the OpenBMB open-source community, and is entirely written by AI with zero human code involvement. The framework adopts the three-stage Forge Engineering methodology (establish standards → bit-for-bit alignment → performance surpassing), achieves training speeds on NVIDIA H100 that exceed Megatron by 10%, has fully run through the pre-training process on Huawei Ascend chips, and has successfully trained the MiniCPM5-1B model.

ForgeTrain's main features

• Production-grade large model pre-training: Supports distributed collaborative training with hundreds to thousands of GPUs, and can be directly put into production use.
• Cross-hardware platform adaptation: Simultaneously supports NVIDIA H100 and Huawei Ascend series chips, and has fully run through the Ascend pre-training process.
• Performance surpassing human implementations: Training speed on H100 is 10% faster than NVIDIA Megatron, saving 10% in compute costs for the same task.
• Bit-for-bit consistency verification: The AI-generated framework can produce completely identical numerical results to the reference implementation (Megatron) under the same inputs, ensuring correctness.
• Automated evaluation Harness: Built-in automated testing and performance evaluation system that turns "runs correctly" and "runs fast" into standards that machines can automatically judge.

ForgeTrain's technical principles

• The three-stage Forge Engineering methodology：
  • Harnessing: Collect key runtime data from Megatron, build an automated evaluation Harness, and define correctness and performance benchmarks.
  • Bit-for-Bit Replication: Under the constraints of the Harness, AI builds a version that is bit-for-bit identical to Megatron, verifying AI's ability to precisely reproduce complex systems.
  • Surpassing: Remove the binary consistency restriction, switch to a performance-oriented Harness, and let AI autonomously iterate in a larger search space, ultimately achieving speed surpassing.
• Code as deeply customized products: Unlike traditional software, which treats code as a long-term maintained "asset," Forge Engineering unbinds code into deeply customized products forged on demand—the same Harness can forge completely different dedicated optimization implementations on different hardware.
• Zero abstraction loss: Generality is preserved in the Harness specifications, while high performance remains in each forging, eliminating the traditional trade-off between generality vs. customization.

How to use ForgeTrain

• Open-source access: The code is fully open source, GitHub address:https://github.com/OpenBMB/ForgeTrain
• Included content: ForgeTrain framework code (including both H100 and Ascend versions) + the complete Agent Harness toolchain (evaluation standards, testing specifications, and agent orchestration solutions).
• Reproduction path: Any team can use Agent Harness to reproduce the entire process from bit-for-bit alignment to performance surpassing.
• Practical verification: ModelBest has already used ForgeTrain to complete the full pre-training of MiniCPM5-1B on Huawei Ascend chips, proving the feasibility of the closed loop.

ForgeTrain's core advantages

• The first production-grade verification of AI making AI: Unlike NVIDIA VibeTensor, which explicitly labels itself "non-production," Anthropic's C compiler, and OpenAI Harness, ForgeTrain is the only AI-generated framework that simultaneously achieves production-grade usability + outstanding performance + full open source.
• 100x increase in R&D efficiency: Transforming the development of large model training frameworks from human-led to AI-autonomous forging.
• Breaking the bottleneck of the domestic computing power ecosystem: No need to spend ten years building another CUDA; AI can instantly forge a dedicated software stack for each chip, achieving "however fast chips ship, software can keep up just as fast."
• Cost savings: Directly saves 10% of computing power costs for equivalent tasks.
• L4-level AI making AI: Reaching L4 on the five-level evolution path of "AI making AI," that is, AI transforming AI's own infrastructure, and recursion begins to compound.

ForgeTrain's project address

• GitHub repository：https://github.com/OpenBMB/ForgeTrain

Comparison of ForgeTrain with similar competitors

ForgeTrain's application scenarios

• Large model pretraining: Directly replaces human-written training frameworks such as Megatron for production-grade large model training.
• Domestic computing power adaptation: Rapidly forges dedicated high-performance training frameworks for domestic chips such as Huawei Ascend, breaking the CUDA ecosystem monopoly.
• AI research acceleration: Reduces large model R&D costs, transforming the annual capability growth of large models from a "function of manpower scale" into a "function of computing power scale."
• Software engineering paradigm transformation: As the first instance of Forge Engineering, it provides methodological reference for the AI automatic generation of other complex systems (compilers, databases, operating systems, etc.).
• On-device model training: ModelBest has already used it to train MiniCPM5-1B, which ranks first globally on the AA leaderboard at the below-2B scale, suitable for efficient on-device model R&D.