SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

JoyAI-LLM-Flash - JD.com's Open-Source Mixture-of-Experts Large Model

Author: AI小集 · Original source: AI工具集

What is JoyAI-LLM-Flash

JoyAI-LLM-Flash is a medium-sized instruction large model open-sourced by JD.com. It adopts a Mixture-of-Experts (MoE) architecture, with a total parameter count of 48B, only 3B activated parameters, and support for a 128K ultra-long context. The model innovatively introduces the FiberPO optimization framework—applying fiber bundle theory to reinforcement learning for the first time, combined with the Muon optimizer for SFT, DPO, and RL training; it also adopts dense Multi-Token Prediction (MTP) technology, increasing throughput by 1.3-1.7 times compared to the non-MTP version. Based on 20 trillion tokens of pre-training data, JoyAI-LLM-Flash performs excellently on tasks such as frontier knowledge understanding, logical reasoning, code generation, and agent interaction, making it suitable for scenarios such as efficient on-device inference, enterprise-level Agent development, and long-text processing.

JoyAI-LLM-Flash

Main Features of JoyAI-LLM-Flash

  • Ultra-Long Context Processing: Supports a 128K token context length, capable of handling long documents, long conversations, and complex multi-turn interaction tasks.
  • Efficient MoE Inference: Total parameter count of 48B, with only 3B activated parameters, 256 experts with dynamic routing, activating 8 experts per token, balancing performance and low inference cost.
  • Multi-Token Prediction (MTP): Adopts dense MTP technology, with throughput increased by 1.3-1.7 times compared to the non-MTP version, significantly improving generation efficiency.
  • Code Generation Capability: Based on 20 trillion tokens of pre-training, skilled in logical reasoning and code generation, supporting assisted development in multiple programming languages.
  • Agent Interaction: Optimized for Agent scenarios, supporting complex task planning, tool calling, and multi-step reasoning.
  • Frontier Knowledge Understanding: Covers knowledge across a wide range of domains, with strong semantic understanding and knowledge question-answering capabilities.
  • Instruction Following Optimization: Through multi-stage training with SFT, DPO, and RL, accurately understands and executes user instructions.

Technical Principles of JoyAI-LLM-Flash

  • Mixture-of-Experts Architecture (MoE): Adopts a sparsely activated MoE design, with a total parameter count of 48B but only 3B parameters activated per token. Through 256 expert modules with dynamic routing (selecting 8 experts per token), it greatly reduces inference cost while maintaining high performance.
  • Dense Multi-Token Prediction (Dense MTP): Innovatively introduces a dense MTP mechanism, predicting multiple future tokens in parallel in a single forward pass, solving the instability problem of traditional models during scale expansion, and increasing throughput by 1.3-1.7 times.
  • FiberPO Optimization Framework: Introduces Fiber Bundle Theory into reinforcement learning optimization for the first time, combined with the Muon optimizer for parameter updates, improving training stability and convergence efficiency.
  • Multi-Stage Training Strategy: Adopts a three-stage training process of SFT (Supervised Fine-Tuning) + DPO (Direct Preference Optimization) + RL (Reinforcement Learning), gradually optimizing the model's instruction-following ability and output quality.
  • MLA Attention Mechanism: Uses Multi-head Latent Attention, with a hidden dimension of 2048, supporting 32 attention heads, efficiently handling long-sequence dependencies.
  • SwiGLU Activation Function: Uses SwiGLU as the non-linear activation, combined with a 129K vocabulary size, improving the model's expressiveness and training stability.

JoyAI-LLM-Flash Project Address

  • HuggingFace Model Repository：https://huggingface.co/jdopensource/JoyAI-LLM-Flash

Application Scenarios of JoyAI-LLM-Flash

  • Efficient On-Device Inference: With only 3B activated parameters, it is suitable for deployment in resource-constrained on-device scenarios such as mobile phones and IoT devices, enabling localized AI assistants.
  • Intelligent Customer Service Systems: Supports a 128K ultra-long context and can handle complex multi-turn conversations, making it suitable for large-scale customer service automation in fields such as e-commerce and finance.
  • Code-Assisted Development: Skilled in logical reasoning and code generation, it can provide developers with real-time code completion, bug fixing, and technical documentation interpretation.
  • Enterprise-Grade Agent Platform: Optimized for agent interaction, it supports task planning, tool invocation, and multi-step reasoning, making it suitable for scenarios such as RPA and automated office work.
  • Long Document Processing: The 128K context capability supports professional scenarios requiring the processing of large amounts of text, such as paper analysis, contract review, and research report summarization.
  • Content Creation Assistance: Based on 20 trillion tokens of pre-training, it can assist in writing commercial content such as marketing copy, product descriptions, and news articles.