What is Apertus
Apertus is Switzerland's first large-scale, open, multilingual large language model, jointly launched by EPFL, ETH Zurich, and the Swiss National Supercomputing Centre (CSCS). It is available in two parameter versions, 70B and 8B, and is trained on large-scale language data, of which 40% is non-English, including Swiss German, Romansh, and other languages previously underrepresented in LLMs. Apertus uses a decoder-only Transformer architecture, based on the new xIELU activation function and the AdEMAMix optimizer. The model is fully open, with model weights, data, and training details available, allowing users to use it on their own servers and maintain control over their data.
Main features of Apertus
-
Text generation: Generates coherent, relevant text based on user-provided prompts.
-
Multilingual support: Supports over 1,811 languages, including many minority languages previously underrepresented in LLMs.
-
Transparency and openness: Model weights, data, and training details are fully public, and users can use it on their own servers.
-
Long-context processing: Supports long-context processing and is suitable for complex tasks.
Technical principles of Apertus
- Model architecture: Apertus uses a dense decoder-only Transformer architecture. The model comes in two scales, 8B and 70B, with 32 layers/32 attention heads and 80 layers/64 attention heads respectively. It uses the xIELU activation function, RMSNorm normalization, RoPE positional encoding, and grouped-query attention to improve model efficiency and long-context processing capability.
- Pretraining objective: It uses the Goldfish objective function, randomly masking some tokens to prevent the model from learning exact context mappings, effectively suppressing verbatim recall while preserving downstream task performance. The pretraining data comes entirely from publicly available sources, respects content owners' opt-out wishes, and avoids using copyrighted, non-licensed, toxic, or personally identifiable information.
- Pretraining data: Apertus is pretrained on over 15 trillion tokens of data, covering more than 1,800 languages. The data sources are diverse, including high-quality web crawl data, code data, math data, and more. Through various filtering mechanisms, such as respecting crawl restrictions in robots.txt files and removing personally identifiable information and toxic content, data compliance is ensured. To improve the model's multilingual capability and data diversity, a large amount of non-English content is allocated in the pretraining data.
- Training process: It is trained using the AdEMAMix optimizer and WSD learning rate scheduling to ensure training stability and efficiency. By gradually increasing the context length, the model can process longer text sequences, supporting contexts of up to 65,536 tokens.
- Post-Training: Through instruction fine-tuning and alignment training, the QRPO algorithm is used to optimize the model's behavior, making it safer, more useful, and more aligned with human values when generating text. The post-training stage enables the model to better understand and generate instruction-following text.
Apertus project address
- Project official website：https://www.swiss-ai.org/apertus
- HuggingFace model repository：https://huggingface.co/collections/swiss-ai/apertus-llm-68b699e65415c231ace3b059
- Technical paper：https://github.com/swiss-ai/apertus-tech-report
Application scenarios of Apertus
- Multilingual dialogue systems: Suitable for building multilingual chatbots, customer service systems, etc., providing users with cross-language communication and information access services.
- Code generation and assistance: Generate code snippets based on natural language descriptions, helping developers quickly implement programming tasks and improve development efficiency, suitable for software development assistance tools.
- Education and learning assistance: Generate educational content, answer academic questions, provide learning suggestions, etc., used in educational scenarios such as online education platforms and intelligent tutoring systems.
- Content creation: Assist in creating text content, such as writing articles, stories, news reports, etc., providing inspiration and writing assistance for content creators.
- Translation services: Used in translation tasks, providing text translation services, and supporting cross-language information dissemination and communication.