What is Instella
Instella is a series of 3-billion-parameter open-source language models introduced by AMD. The models are trained entirely from scratch on AMD Instinct™ MI300X GPUs, based on an autoregressive Transformer architecture, containing 36 decoder layers and 32 attention heads, and supporting sequences of up to 4096 tokens. Instella undergoes multi-stage training, including large-scale pretraining, supervised fine-tuning, and preference optimization, enhancing natural language understanding, instruction following, and dialogue capabilities. Instella surpasses existing open-source models on multiple benchmarks and is competitive with state-of-the-art open-weight models. AMD fully open-sources Instella's model weights, training configurations, datasets, and code to promote collaboration and innovation in the AI community.
Instella's Main Features
- Natural Language Understanding: Understands complex natural language text and handles various language tasks, such as question answering, text generation, and semantic analysis.
- Instruction Following: Based on supervised fine-tuning (SFT) and direct preference optimization (DPO), accurately understands and executes user instructions, generating responses that align with human preferences.
- Multi-Turn Dialogue Capability: Supports multi-turn interaction and conducts coherent conversations based on context.
- Problem-Solving Ability: Performs excellently on tasks such as math problems, logical reasoning, and knowledge question answering.
- Multi-Domain Adaptability: Based on diverse training data, adapts to multiple domains, such as academia, programming, mathematics, and everyday conversation.
Instella's Technical Principles
- Transformer Architecture: Based on an autoregressive Transformer architecture, containing 36 decoder layers, each with 32 attention heads, and supporting a maximum sequence length of 4096 tokens.
- Efficient Training Techniques: FlashAttention-2, Torch Compile, and bfloat16 mixed-precision training, optimizing memory usage and computational efficiency.
- Multi-Stage Training: Uses 4.065 trillion tokens for large-scale pretraining to establish foundational language understanding capabilities. Further trains on the basis of the first stage, using an additional 57.575 billion tokens, to enhance task-specific capabilities.
- Supervised Fine-Tuning (SFT): Fine-tunes with high-quality instruction-response pair data to improve instruction-following capability.
- Direct Preference Optimization (DPO): Optimizes the model based on human preference data, making outputs more aligned with human values.
- Distributed Training: Based on Fully Sharded Data Parallel (FSDP) technology, shards model parameters, gradients, and optimizer states within nodes, and replicates them across nodes, enabling large-scale cluster training.
- Datasets: Trained on diverse, high-quality datasets, including academic, programming, mathematics, and dialogue data, as well as synthetic datasets, ensuring the model has broad knowledge and capabilities.
Instella's Project Address
- Project Website：https://rocm.blogs.amd.com/artificial-intelligence/introducing-instella
- GitHub Repository：https://github.com/AMD-AIG-AIMA/Instella
- HuggingFace Model Hub：https://huggingface.co/collections/amd/instella
Instella's Application Scenarios
- Intelligent Customer Service: Automatically answers questions, provides personalized services, and improves customer experience.
- Content Creation: Generates copy, stories, etc., helping content creators improve efficiency.
- Educational Tutoring: Answers academic questions, provides study suggestions, and assists students in learning.
- Programming Assistance: Generate code snippets, provide programming advice, and help developers solve problems.
- Enterprise Knowledge Management: Integrate company knowledge, provide internal consulting, and improve collaboration efficiency.