In June 2021, the Beijing Academy of Artificial Intelligence (BAAI) launched Wudao 2.0, a successor version to Wudao 1.0, as China's first ultra-large-scale intelligent model system. Wudao is a language model designed to surpass OpenAI's GPT-3 and Google's LaMDA in human-level thinking. After being trained on 4.9TB of images and text and surpassing the state-of-the-art (SOTA) level on 9 benchmarks, Wudao is closer than any peer to achievingartificial general intelligence (AGI)and human-level thinking.
Wudao was trained on 4.9 TB of high-quality English and Chinese images and text:
- 1.2TB Chinese text data
- 2.5TB Chinese graphic data
- 1.2TB English text data
Wudao is trained based on the open-source MoE system FastMoE. MoE is amachine learningtechnique that works as follows:
Divide the predictive modeling task into subtasks, train an expert (learner) model for each subtask, develop a gating model that learns which expert to consult based on the input to be predicted, and combine the predictions. FastMoE enables Wudao to consult different expert models in parallel and switch to the model with the best prediction result. For example, if the input is English text, Wudao will use the prediction model that can generate responses in English text.
