MaxText is a high-performance, highly scalable open-source large language model (LLM) library and reference implementation, written in pure Python/JAX, primarily targeting training on Google Cloud TPU and GPU. The project recommends using the latest PyPI release or the corresponding prebuilt container image; the main branch is not guaranteed to be production-ready; Python 3.12 is recommended, and other versions may encounter compatibility issues.
[Purpose] MaxText provides a high-performance model library and demonstrates how to perform pre-training or post-training with high performance and at large scale. It can serve both as a reference implementation for research experiments, ideation, and inspiration, and for building model training from scratch, such as small dense models like Llama 8B or large MoE models like DeepSeek-V3. The project's goal is to make it easy for users to explore (a)*(b) combinations by providing multiple models (dimension a) and multiple techniques (dimension b), so as to efficiently train models for specific use cases.
[Main Capabilities]
- Supports pre-training (scalable to tens of thousands of chips) and scalable post-training, covering supervised fine-tuning (SFT), group relative policy optimization (GRPO), and group sequence policy optimization (GSPO), among other reinforcement learning techniques.
- From a single host to ultra-large-scale clusters, it maintains high Model FLOPs Utilization (MFU) and tokens/second while remaining simple and largely free of manual optimization, thanks to the JAX and XLA compilers.
- Leverages the JAX AI ecosystem: Flax (neural networks), Tunix (post-training), Orbax (checkpointing), Optax (optimization), Grain (data loading).
- In addition to text-only LLMs, it also supports multimodal training for Gemma 3, Gemma 4, and Llama 4 VLM.
- Provides reference implementations that achieve optimal performance across multiple dimensions such as sharding, quantization, and checkpointing.
[Supported Models]
- Google: Gemma 4 (26B MoE, 31B Dense), Gemma 3 (4B, 12B, 27B), Gemma 2 (2B, 9B, 27B), Gemma 1 (2B, 7B)
- Alibaba: Qwen 3.5 MoE (35B, 397B), Qwen 3 Next (80B), Qwen 3 MoE (30B, 235B), Qwen 3 MoE 2507 (235B, 480B), Qwen 3 Dense (0.6B, 1.7B, 4B, 8B, 14B, 32B), Qwen 2.5 (1.5B, 7B, 14B)
- DeepSeek AI: DeepSeek V3.2 (671B), DeepSeek V3.1 (671B), DeepSeek V3 0324 (671B), DeepSeek R1 0528 (671B), DeepSeek V2 (16B, 236B)
- Moonshot AI: Kimi K2 (1T), Kimi K2-Thinking (1T), Kimi K2.5 (text), Kimi K2.6 (text)
- Meta: Llama 4 Scout (109B) and Maverick (400B), Llama 3.3 (70B), 3.1 (8B, 70B, 405B), 3.0 (8B, 70B, 405B), Llama 2 (7B, 13B, 70B)
- OpenAI: GPT-OSS (20B, 120B), GPT3 (52K, 6B, 22B, 175B)
- Mistral AI: Mixtral (8x7B, 8x22B), Mistral (7B)
- For diffusion models, see MaxDiffusion (LTXV, Wan 2.1, Flux, SDXL, etc.)
[Usage] MaxText can be installed from PyPI via pip, or you can use a prebuilt container image. The project provides a Read The Docs documentation site and getting started guides, including SFT and RL tutorials on single-host/multi-host TPU. It supports running in a decoupled mode with no GCP dependencies. If you want to build a model from scratch, you can fork and modify MaxText to train your own model, and build the most efficient model on TPU or GPU through configuration and model design. Post-training supports proprietary or open-source models, using Tunix to provide a scalable framework; for RL (such as GRPO), it uses vLLM for sampling and plans to support multi-host via Pathways.
[Applicable Scenarios] Suitable for research experiments, production-grade LLM projects, large-scale pre-training and post-training, multimodal model training, and developers who want to use the reference implementation as a basis for exploring model design.
[Limitations] The main branch is not guaranteed to be production-ready; Python 3.12 is recommended, and other versions may have compatibility issues; the project is primarily aimed at Google Cloud TPU and GPU. Details such as specific fees, pricing, and platform support are not yet provided in the materials.
[Latest Updates Summary] Recent updates include: JEV-style decision scoring supported on TPU via vLLM; Streaming DiLoCo distributed low-communication training; DeepSeek V4 Flash (284B) support; Tokamax GMM v2 MoE kernel; RL support for Qwen3 30B and GPT-OSS 20B; migration to Flax NNX; Qwen3.5 35B & 397B support; new evaluation framework; Kimi-K2 series support; GEPA prompt optimization notebook; MuonClip optimizer; DeepSeek-V3.2 with sparse attention; Gemma 4 multimodal; DeepSeek Engram and mHC; Qwen3-Next support; Muon optimizer; DeepSeek V3.1; RL/SFT Notebook tutorials; GSPO/GRPO multi-host support; vocabulary chunking; GPT-OSS support; PyPI package release; src layout migration, etc.
