What is nanochat
nanochat is an open-source project released by AI expert Andrej Karpathy, which trains small language models with extremely low cost and an efficient pipeline to achieveChatGPT-like conversational capabilities.For only about $100 (using 8 H100 GPUs for 4 hours of training), you can train a small model capable of basic conversation, creating stories/poems, and answering simple questions. If the budget is increased to $1000 (about 41.6 hours of training), model performance can improve significantly, enabling it to solve simple math/code problems and participate in multiple-choice question tests.The project includes a complete pipeline from data preparation, pretraining, midtraining, supervised fine-tuning (SFT), reinforcement learning (RL), to inference deployment, with end-to-end training implemented in about 8,000 lines of code. The code is concise and easy to read, suitable for learning and practice.
Main features of nanochat
-
Tokenizer training: Implemented in Rust to train the tokenizer, responsible for converting text into a sequence of symbol codebook tokens.
-
Pretraining: Pretrain a Transformer-architecture large language model on the FineWeb dataset, and evaluate model performance through the CORE metric.
-
Midtraining: Perform midtraining on the SmolTalk user-assistant dialogue dataset, multiple-choice question datasets, and tool-use datasets to adapt the model to conversational scenarios.
-
Supervised fine-tuning (SFT): Perform supervised fine-tuning on world knowledge multiple-choice question datasets (ARC-E/C, MMLU), math datasets (GSM8K), and code datasets (HumanEval) to improve the model's performance on specific tasks.
-
Reinforcement learning fine-tuning (RL): Use the "GRPO" algorithm to perform reinforcement learning fine-tuning on the GSM8K dataset, further optimizing model performance.
-
Inference deployment: Implement efficient model inference, supporting KV cache, a simple prefill/decode process, tool use (a Python interpreter in a lightweight sandbox environment), and interaction with the model through a CLI or a ChatGPT-like WebUI.
-
Report card generation: Generate a single Markdown-format report card summarizing the entire training and inference pipeline, and present the results in a "gamified" form.
Technical principles of nanochat
-
Minimalist code architecture: The entire project is only about 8,000 lines of code, implemented in a single codebase, with very few dependencies and a clear structure, making it easy to understand and modify.
-
Rust language tokenizer: Implemented in Rust to train the tokenizer, responsible for converting text into a sequence of symbol codebook tokens, improving tokenization efficiency and performance.
-
Transformer architecture: Build a large language model based on the Transformer architecture, learning language patterns and knowledge through pretraining.
-
Data-driven training: Perform pretraining on datasets such as FineWeb, allowing the model to learn language expression and knowledge through large amounts of text data.
-
Midtraining adaptation: Perform midtraining on dialogue datasets such as SmolTalk to adapt the model to conversational scenarios and specific tasks.
-
Reinforcement learning optimization: Use the "GRPO" algorithm to perform reinforcement learning fine-tuning on specific datasets, further optimizing model performance.
-
Efficient inference engine: Implement an inference engine with KV cache, supporting prefill and decode processes, improving inference efficiency.
-
WebUI interaction: Provides a ChatGPT-like web interface, allowing users to interact with the trained model through the WebUI.
nanochat project address
- Github repository：https://github.com/karpathy/nanochat
nanochat application scenarios
-
Individuals and teams: Suitable for individuals or teams with strong cybersecurity awareness to quickly establish encrypted communication channels within an internal network.
-
Developers and technology enthusiasts: A practical platform for learning and researching P2P networks, encryption technology, and command-line application development.
-
Temporary work groups: Such as emergency response teams, to quickly form a communication network without a central server.
-
Education and research: nanochat provides researchers and learners with a low-cost, easy-to-understand and improve LLM development platform.