Llama 3

What is Llama 3

Llama 3 is Meta's latest newly open-sourced next-generation large language model (LLM), including models with two parameter scales, 8B and 70B, marking another major advancement in the field of open-source artificial intelligence. As the third-generation product in the Llama series, Llama 3 not only inherits the powerful capabilities of previous-generation models, but also provides more efficient and more reliable AI solutions through a series of innovations and improvements, aiming to support a wide range of application scenarios through advanced natural language processing technology, including but not limited to programming, problem solving, translation, and dialogue generation.

The series of Llama 3 models

Llama 3 currently offers two models, the 8B (8 billion parameters) and 70B (70 billion parameters) versions. These two models are designed to meet application needs at different levels, providing users with flexibility and freedom of choice.

• Llama-3-8B: 8B parameter model, which is a relatively smaller but efficient model with 8 billion parameters. It is designed for application scenarios that require fast inference and fewer computing resources, while maintaining a high performance standard.
• Llama-3-70B: 70B parameter model, which is a larger-scale model with 70 billion parameters. It can handle more complex tasks and provide deeper language understanding and generation capabilities, making it suitable for applications with higher performance requirements.

Subsequently, Llama 3 will also release a model with a 400B parameter scale, which is currently still in training. Meta also stated that after completing the training of Llama 3, it will also publish a detailed research paper.

Llama 3's official website entry

• Official project homepage：https://llama.meta.com/llama3/
• GitHub model weights and code：https://github.com/meta-llama/llama3/
• Hugging Face model：https://huggingface.co/collections/meta-llama/meta-llama-3-66214712577ca38149ebb2b6

Where Llama 3 has improved

• Parameter scale: Llama 3 provides models with two parameter scales, 8B and 70B. Compared with Llama 2, the increase in the number of parameters enables the model to capture and learn more complex language patterns.
• Training dataset: Llama 3's training dataset is 7 times larger than Llama 2, containing more than 15 trillion tokens, including 4 times the code data, which makes Llama 3 more outstanding in understanding and generating code.
• Model architecture: Llama 3 adopts a more efficient tokenizer and Grouped Query Attention (GQA) technology, improving the model's inference efficiency and ability to process long texts.
• Performance improvements: Through improved pre-training and post-training processes, Llama 3 has made progress in reducing the false refusal rate, improving response alignment, and increasing model response diversity.
• Safety: New trust and safety tools such as Llama Guard 2 were introduced, as well as Code Shield and CyberSec Eval 2, enhancing the model's safety and reliability.
• Multilingual support: Llama 3 added high-quality non-English data in more than 30 languages to its pre-training data, laying the foundation for future multilingual capabilities.
• Reasoning and code generation: Llama 3 has demonstrated greatly improved capabilities in areas such as reasoning, code generation, and instruction following, making it more precise and efficient in handling complex tasks.

Llama 3's performance evaluation

According to Meta's official blog, the instruction-tuned Llama 3 8B model outperforms models of comparable parameter scale (Gemma 7B, Mistral 7B) on benchmarks such as MMLU, GPQA, HumanEval, GSM-8K, and MATH, while the fine-tuned Llama 3 70B also outperforms similarly sized Gemini Pro 1.5 and Claude 3 Sonnet models on benchmarks including MLLU, HumanEval, and GSM-8K.

In addition, Meta has developed a new high-quality human evaluation set containing 1,800 prompts covering 12 key use cases: seeking advice, brainstorming, classification, closed-ended question answering, coding, creative writing, extraction, embodying a character/persona, open-ended question answering, reasoning, rewriting, and summarization. Through comparison with competing models such as Claude Sonnet, Mistral Medium, and GPT-3.5, human evaluators performed preference rankings based on this evaluation set, and the results show that Llama 3's performance in real-world scenarios is excellent, with at least a 52.9% win rate.

Technical architecture of Llama 3

• Decoder architecture: Llama 3 adopts a decoder-only architecture, which is a standard Transformer model architecture mainly used for handling natural language generation tasks.
• Tokenizer and vocabulary size: Llama 3 uses a tokenizer with 128K tokens, which enables the model to encode language more efficiently, thereby significantly improving performance.
• Grouped Query Attention (GQA): To improve inference efficiency, Llama 3 adopts GQA technology in both the 8B and 70B models. This technology reduces computation by grouping queries in the attention mechanism while maintaining model performance.
• Long sequence processing: Llama 3 supports sequences up to 8,192 tokens, using masking technology to ensure that self-attention does not cross document boundaries, which is especially important for processing long texts.
• Pre-training dataset: Llama 3 was pre-trained on more than 15TB of tokens. This dataset is not only huge in scale but also high in quality, providing the model with rich language information.
• Multilingual data: To support multilingual capabilities, Llama 3's pre-training dataset contains more than 5% non-English high-quality data, covering more than 30 languages.
• Data filtering and quality control: The Llama 3 development team developed a series of data filtering pipelines, including heuristic filters, NSFW (not safe for work) filters, semantic deduplication methods, and text classifiers, to ensure the high quality of the training data.
• Scalability and parallelization: The training process of Llama 3 adopted data parallelism, model parallelism, and pipeline parallelism. The application of these technologies enables the model to be trained efficiently on a large number of GPUs.
• Instruction Fine-Tuning: On the basis of the pre-trained model, Llama 3 further improves the model's performance on specific tasks, such as dialogue and programming tasks, through instruction fine-tuning.

How to use Llama 3

Developers

Meta has open-sourced its Llama 3 model on GitHub, Hugging Face, and Replicate. Developers can use tools such as torchtune to customize and fine-tune Llama 3 to adapt it to specific use cases and needs. Interested developers can check the officialgetting started guideand go download and deploy it.

• Official model download：https://llama.meta.com/llama-downloads
• GitHub address：https://github.com/meta-llama/llama3/
• Hugging Face address：https://huggingface.co/meta-llama
• Replicate address：https://replicate.com/meta

Ordinary users

Ordinary users who do not understand technology and want to experience Llama 3 can use it in the following ways:

• Visit Meta's newly launchedMeta AIchat assistant to experience it (Note: Meta.AI is region-locked and can only be used in some countries)
• Visit Chat with Llama provided by Replicate to experience ithttps://llama3.replicate.dev/
• Use Hugging Chat (https://huggingface.co/chat/), and you can manually switch the model to Llama 3