What is Fox-1
Fox-1 is a series of small language models (SLMs) launched by TensorOpera, including Fox-1-1.6B and Fox-1-1.6B-Instruct-v0.1. The Fox-1 model is pre-trained on 3 trillion web-crawled document data and fine-tuned on 5 billion instruction-following and multi-turn dialogue data. Fox-1 adopts a 3-stage data curriculum and deep architecture design, featuring a 256K extended vocabulary and GQA mechanism, improving efficiency and performance. The Fox-1 model demonstrates outstanding performance on multiple standard language model benchmarks including ARC Challenge, HellaSwag, MMLU, GSM8k, etc., surpassing models with twice its parameter size.
Main Features of Fox-1
- Text Generation and Understanding: Fox-1 can handle and generate various text tasks, such as text summarization, translation, question answering, etc.
- Instruction Following: Fox-1-1.6B-Instruct-v0.1 is specifically fine-tuned for instruction-following tasks, capable of understanding and executing users' direct instructions.
- Multi-turn Dialogue: The model is fine-tuned on multi-turn dialogue data and can be used in dialogue systems, providing coherent and relevant responses.
- Long Context Processing: By adopting Rotary Positional Embeddings (RoPE) and a 3-stage data curriculum, Fox-1 can effectively process sequences up to 8K, suitable for handling long documents and lengthy texts.
- High-Efficiency Inference: Fox-1 achieves inference speed and throughput comparable to larger models while maintaining a smaller model size.
Technical Principles of Fox-1
- 3-Stage Data Curriculum: Fox-1's pre-training includes a novel 3-stage data curriculum, gradually increasing the chunk length of training samples from 2K to 8K, optimizing long context processing capability.
- Deep Architecture Design: Fox-1 adopts a deep architecture with 32 self-attention layers, deeper than other models, enhancing the model's reasoning ability.
- Grouped Query Attention (GQA): Query heads are grouped, with each group sharing the same key-value heads, improving training and inference speed and reducing memory usage.
- Shared Input-Output Embeddings: Fox-1 shares input and output embedding layers, reducing the number of model parameters and improving weight utilization.
- Extended Vocabulary: The model uses a 256K vocabulary, which compared to standard vocabulary can more effectively encode information, reduce the probability of unknown words, and improve downstream task performance.
- Pre-normalization: Fox-1 uses RMSNorm for pre-normalization, helping to improve training efficiency.
- RoPE Positional Encoding: Fox-1 adopts RoPE, helping to encode relative positional dependencies between tokens.
Project Address of Fox-1
- Project Official Website：tensoropera-unveils-fox
- HuggingFace Model Repository：
- arXiv Technical Paper：https://arxiv.org/pdf/2411.05281
Application Scenarios of Fox-1
- Chatbots and Customer Service: Build chatbots, provide customer consultation services, handle multi-turn dialogues, and answer user questions.
- Content Creation and Editing: In the content industry, help generate creative copy, edit and polish articles, and provide writing suggestions.
- Language Translation: Applied in the field of machine translation, helping users with translation work between languages.
- Education and Learning: In the field of education, as a teaching aid, it provides language learning support, including grammar checking, writing tutoring, etc.
- Information retrieval and question-answering systems: Integrated into search engines and question-answering systems to provide fast and accurate information retrieval and answer generation.