llama3-from-scratch

llama3-from-scratch is an educational open-source project aimed at implementing the Llama 3 model from scratch, using a tensor-by-tensor, matrix-multiplication-by-matrix-multiplication approach to give readers a deep understanding of the inner workings of large language models. The project was developed by naklecha, and the code is hosted on GitHub.

Purpose

The project is mainly for learning and research, helping developers understand the inference process of Llama 3, including key steps such as model loading, tokenization, embedding, attention mechanisms, positional encoding, and feed-forward networks. It does not rely on the neural network modules of high-level frameworks such as PyTorch or TensorFlow (it only uses built-in tensor operations); instead, it manually implements all computations, thereby revealing the underlying details.

Main capabilities

• Read tensors directly from the model files provided by Meta, without relying on model class definitions.
• Use tiktoken for text tokenization (the author mentions that a BPE tokenizer was not implemented, but provides a link to Andrej Karpathy's minbpe as a reference).
• Implement RMS normalization and explain the role of norm_eps.
• Implement rotary position embedding (RoPE), rotating query and key vectors via complex dot products.
• Implement the multi-head attention mechanism, including unpacking and computing queries, keys, and values, as well as the causal mask.
• Implement the feed-forward network and build the Transformer layer by layer (32 layers in total).
• Finally decode the embedding and predict the next token.

How to use

1. Download the Llama 3 weights: you need to obtain them from Meta's official link (https://llama.meta.com/llama-downloads/）下载模型权重，并遵守相关许可。
2. Environment preparation: dependencies such as tiktoken need to be installed (the specific dependencies are not listed in the materials, but tiktoken is used in the code).
3. Run the code: the project is provided as a Jupyter Notebook or Python script. Execute each cell step by step and observe the tensor shapes and computation results.
4. The code comments in detail the shape changes at each step, and readers are advised to follow the comments to understand it.

Applicable scenarios

• Deep learning researchers or students who want to understand the implementation of Transformer and Llama 3 from the ground up.
• Developers who need to customize or modify the inference process of Llama 3 and learn how to manually operate tensors.
• Educational use, as teaching material demonstrating the internal computations of large language models.

Limitations

• The project only implements inference and does not involve training.
• Users need to download the Llama 3 weights themselves, and the weights are subject to Meta's license restrictions.
• A BPE tokenizer is not implemented; it relies on tiktoken.
• The code may not be optimized and is not suitable for production deployment.
• The materials do not provide specific installation commands, dependency lists, or runtime environment requirements, so users need to infer them from the code.

Other information

The author encourages feedback and support, and provides Twitter and Buy Me a Coffee links. The project belongs to the A10 research lab, aiming to make research more accessible.