nanowhale is a language model project open-sourced by Hugging Face on GitHub. Its core is a language model with approximately 110 million parameters, trained entirely from scratch using the DeepSeek-V4 architecture. The repository provides all the code, configuration files, and tokenizer needed to pretrain and fine-tune the model, making it convenient for researchers to reproduce or conduct experiments on top of it.
Architecturally, the project implements the full feature set of DeepSeek-V4 at a tiny scale. Multi-head Latent Attention (MLA) uses 8 attention heads, 1 KV head (i.e., MQA), head_dim of 96 (of which 32 dimensions are used for RoPE and 64 for NoPE), and q_lora_rank of 160. Mixture of Experts (MoE) contains 4 routed experts and 1 shared expert, uses a top-2 routing strategy, and the SwiGLU FFN has a dimension of 640. In addition, the model also implements Hyper-Connections with hc_mult=4 and uses Sinkhorn routing (2 iterations). Multi-Token Prediction (MTP) contains 1 next-token prediction layer.
In terms of training, the repository provides the results of pretraining and SFT (supervised fine-tuning). Pretraining was performed for 5,000 steps on the FineWeb-Edu dataset, and SFT was performed for 3,000 steps on the SmolTalk dataset. The evaluation metric is perplexity on held-out English text.
The model has two limitations that need attention. First, under bf16 precision, the model produces NaN (not a number), because the values produced by the Hyper-Connections architecture exceed the representable range of bf16, so both inference and training should use fp32 precision. Second, due to the custom architecture, the from_pretrained method will reinitialize some weights, and the official recommendation is to load weights manually using load_state_dict (see the model card for a specific example). In addition, the embedding table of the 129K vocabulary occupies 37% of all parameters, which to some extent limits the model's language modeling capacity; this is also an inherent constraint brought by pairing a small model with a large vocabulary.
Overall, nanowhale is suitable for researchers or developers who are interested in the DeepSeek-V4 architecture and want to learn and experiment with it on a small-scale model. In terms of usage, you need to clone the repository and refer to the code, configuration, and model card within it for training or inference, taking care to follow the requirements of fp32 precision and manual weight loading.
