Naive-N0.5-Flash - NaiveAI's first open-source MoE large model

What is Naive-N0.5-Flash

Naive-N0.5-Flash is NaiveAI's first open-source MoE large model, positioned forprogrammingand AI R&D scenarios. Naive-N0.5-Flash is built on Xiaomi's MiMo-V2.5 base model, features a 309B MoE architecture, adopts SWA+DSA hybrid sparse attention, natively supports 1M context, and has no full attention layers. The model was developed with deep AI participation, and its accompanying inference system NaiveRT achieves a single-stream decoding peak of 2122 tokens/s.

Main features of Naive-N0.5-Flash

• Long-context programming: natively supports 1M token context, and can handle ultra-long code repositories and long-horizon software engineering tasks.
• AI R&D assistance: can read papers, reproduce methods, run experiments, and analyze results, directly participating in the entire model R&D workflow.
• Inference acceleration: comes with the NaiveRT inference system, with 50 tokens/s in standard mode and a single-stream decoding peak of 2,122 tokens/s in blazing mode.
• Autonomous research: given a target, compute budget, and evaluation protocol, it can autonomously advance research end to end.

Technical principles of Naive-N0.5-Flash

• SWA–DSA hybrid sparse attention: the model is based on the MiMo-V2.5 base model, replacing the originally expensive global attention layers with DeepSeek sparse attention, alternately stacked with sliding window attention at a 5:1 ratio, with no full attention layers anywhere in the network. SWA handles a 128-token local window, while DSA uses a lightweight indexer to score the full history and select the top-2048 tokens for the backbone to compute.
• Lightweight indexer design: changes DSA's original MLA to GQA, reducing the indexer query heads from 64 to 16, cutting the index selection path latency by 44%, while maintaining undiminished performance on 1M-context Agent tasks.
• Three-stage training pipeline: first freeze the backbone on 50B tokens and align the indexer with the full attention distribution using a KL divergence loss; then switch to sparse attention for 3T tokens of continued pretraining, adapting to the new information selection mechanism and strengthening programming and R&D capabilities; finally 200B tokens of supervised fine-tuning with learning rate decay. The 1M context is maintained throughout, and AI automatically completes top-k recall analysis and numerical stability fixes.
• Hybrid sequence parallelism and memory optimization: to address the difference that SWA only needs a local window while DSA needs the full history, Halo sequence parallelism is used separately; on the memory side, activation offloading granularity is refined to the operator level, and top-k indices are explicitly retained to avoid recomputation changing the selection, allowing 1T tokens to be trained in about 4 days on 512 GPUs.
• NaiveRT inference optimization: through giant operator fusion, PDL enabling upstream and downstream kernels to execute in an overlapping manner, and the speculative decoding loop running entirely on the GPU, single-round decoding latency is reduced from 12.3ms to 3.4ms; it also supports bitwise deterministic sampling by random seed, ensuring consistency between sampling and training logprobs in RL training. All optimizations use the end-to-end real model path as the admission standard.

Follow on WeChat and reply “open source” to join theAI open-source project exchange group

How to use Naive-N0.5-Flash

• Confirm hardware: requires an NVIDIA GPU that supports FP8, the model weights are about 315GB, and additional VRAM needs to be reserved.
• Obtain weights: pull from Hugging Face NaiveAI/Naive-N0.5-Flash-FP8 repository.
• Install dependencies: run pip install "transformers[torch,kernels]>=5.17.0"。
• Load model: use AutoTokenizer and AutoModelForCausalLM with trust_remote_code=True、device_map="auto" method to load.
• Build input: via apply_chat_template pass in the conversation messages and generate the input tensor.
• Generate result: call model.generate() to generate the reply, then use tokenizer.decode() to decode the output.
• Set parameters: for general scenarios, temperature=1.0 and top_p=0.95 are recommended.
• Reproduce high-performance inference: if you need 2,122 tokens/s speed, you can obtain the NaiveRT open-source code, Docker Compose environment, and accompanying benchmark scripts to deploy it yourself.

Core advantages of Naive-N0.5-Flash

• Ultra-long context: natively supports a 1M token context, and there are no full attention layers across the entire network, so long-sequence decoding overhead is far lower than conventional architectures.
• Top-tier inference speed: the accompanying NaiveRT system reaches a peak single-stream decoding speed of 2,122 tokens/s, and speculative decoding latency is reduced by 72.4% compared with SGLang.
• Outstanding R&D capability: PaperBench 63.2, MLE-bench-30 73.7%, NanoGPT SpeedRun 73.8s, surpassing models of the same tier in multiple AI R&D evaluations.
• AI-native R&D: the model has AI participating in architecture design and system optimization, realizing the "AI R&D AI" paradigm in which humans set goals and AI executes iterations.
• Autonomous research potential: given a goal and budget, it can autonomously complete research end-to-end, training the AutoWM world model in 400 hours and topping WorldArena.

Project address of Naive-N0.5-Flash

• Project official website：https://naive.ai/en/research/
• GitHub repository：https://github.com/NaiveAI-Labs/Naive-N0.5-Flash
• HuggingFace model library：https://huggingface.co/NaiveAI/Naive-N0.5-Flash

Comparison of Naive-N0.5-Flash with similar competitors

Application scenarios of Naive-N0.5-Flash

• AI R&D assistant: assists researchers in reading papers, reproducing methods, running experiments, and analyzing results, such as PaperBench and MLE-bench-style R&D tasks.
• Long-horizon software engineering: with native 1M context, it handles large code repositories, completing cross-file understanding, bug fixing, and repository-level code generation.
• Long-context Agent tasks: Maintain the complete task history in long interaction scenarios such as terminal operations and multi-turn tool calls.
• RL training infrastructure: Combined with NaiveRT's high single-stream decoding speed and bitwise deterministic sampling, accelerate long-horizon reinforcement learning rollouts and reduce straggler samples.