SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

ERNIE-4.5-VL - Baidu Wenxin's new-generation open-source multimodal AI model

Author: AI小集 · Original source: AI工具集

What is ERNIE-4.5-VL

ERNIE-4.5-VL is Baidu Wenxin's new-generation open-source multimodal AI model. Based on the ERNIE-4.5-VL-28B-A3B architecture with 3B activated parameters, it focuses on visual language understanding and cross-modal reasoning. The model strengthens semantic alignment through training on massive visual-language data and adopts multimodal reinforcement learning to improve stability. It adds visual grounding and "image thinking" features, and supports scenarios such as document parsing and video analysis. The model supports 2-Bit lossless quantization and a 128K ultra-long context window, optimizes memory usage and inference speed, and can handle ultra-long text content.

ERNIE-4.5-VL

Main features of ERNIE-4.5-VL

  • Multimodal understanding and generation: ERNIE-4.5-VL can process text and image information simultaneously, enabling image caption generation, image-text question answering, image classification, and other functions, and supports comprehensive understanding and creation of multimodal content.
  • Document and chart understanding: The model has powerful document parsing capabilities and can process complex charts, flowcharts, and more. It is widely used in engineering drawing understanding and data analysis in fields such as manufacturing and scientific research.
  • Intelligent reasoning and decision-making: It supports complex multimodal reasoning tasks such as visual question answering and logical reasoning, and can combine text and image information for in-depth analysis and decision-making.
  • Multilingual interaction: It supports interaction in more than 100 languages and is suitable for cross-language multimodal tasks such as multilingual image captioning and cross-language image-text question answering.
  • Long text processing: It has an ultra-long context window (supporting up to 128K), can process large-scale text content, and is suitable for long document analysis and complex task processing.
  • Image thinking feature: It supports tool invocation such as image zoom-in and search, enhancing the model's interactivity and flexibility in visual tasks.
  • Efficient deployment and optimization: Through 2-Bit quantization technology, it significantly reduces memory usage and improves inference speed, supports single-card deployment, and is suitable for resource-constrained scenarios.

Model versions of ERNIE-4.5-VL

  • ERNIE-4.5-21B-A3B-Thinking
    • Parameter scale: 21 billion parameters, with 3 billion parameters activated each time.
    • Features: Supports thinking mode, has stronger reasoning capabilities, and is suitable for handling complex multimodal tasks.
    • Context length: Supports 128k context length.
  • ERNIE-4.5-VL-28B-A3B
    • Parameter scale: 28 billion parameters, with 3 billion parameters activated each time.
    • Architecture: It adopts a heterogeneous mixture-of-experts architecture (MoE), including text, vision, and shared experts, and improves visual task performance through a modality-isolated routing mechanism.
    • Performance: It performs excellently in multimodal tasks, especially in visual perception and document and chart understanding.
    • Context length: Supports 32k context length.
  • ERNIE-4.5-Turbo-VL-Preview
    • Features: Capabilities such as image understanding, creation, translation, and code have been significantly improved. It supports a 32k context length for the first time, and the first-token latency is significantly reduced.
    • Context length: Supports 16k context length.
  • ERNIE-4.5-Turbo-VL-32K-Preview
    • Features: Further improves performance on the basis of ERNIE-4.5-Turbo-VL-Preview and supports longer context lengths.
    • Context Length: Supports a 32k context length.
  • ERNIE-4.5-VL-424B-A47B
    • Parameter Scale: 424 billion parameters, with 47 billion activated parameters.
    • Performance: Stands out in multimodal evaluation benchmarks, with particularly clear advantages on highly difficult reasoning tasks.
    • Features: Supports thinking mode, has strong reasoning capabilities, and also maintains good results on perception tasks.
  • ERNIE-4.5-0.3B
    • Parameter Scale: Only 300 million parameters.
    • Features: A lightweight model suitable for deployment on edge devices, with fast inference speed and performance close to large models.

Technical Principles of ERNIE-4.5-VL

  • Heterogeneous Mixture-of-Experts Architecture (MoE): ERNIE-4.5-VL adopts a heterogeneous mixture-of-experts architecture, dividing parameters into text experts, vision experts, and shared experts. This architecture allows the model to dynamically select the optimal expert combination when handling tasks of different modalities, improving computational efficiency and performance.
  • Modality-Isolated Routing Mechanism: By separating the processing paths for text and images and introducing router orthogonality loss and multimodal balance loss, the model achieves decoupled training of text and visual modalities, significantly improving performance on vision tasks.
  • 2-Bit Lossless Quantization Technology: It adopts the "convolutional code quantization" algorithm to achieve lossless inference at 2-bit precision. This enables the model to greatly reduce memory usage and improve inference speed while maintaining performance, supporting more efficient deployment.

Project Addresses for ERNIE-4.5-VL

  • Github Repository：https://github.com/PaddlePaddle/ERNIE
  • HuggingFace Model Library：https://huggingface.co/collections/baidu/ernie-45
  • Technical Paper：https://yiyan.baidu.com/blog/publication/ERNIE_Technical_Report.pdf

Application Scenarios of ERNIE-4.5-VL

  • Document and Chart Understanding: Used to parse complex engineering drawings, scientific charts, and business documents, helping users quickly extract key information, and applied in fields such as manufacturing, scientific research, and finance.
  • Smart Meter Operations and Maintenance: Embedded in smart meter devices, it generates text descriptions of power usage anomalies in real time and reports faults, improving the efficiency and accuracy of operations and maintenance.
  • Multilingual Interaction: Supports interaction in more than 100 languages, suitable for cross-language image-text question answering, image description, and other tasks, helping global applications.
  • Intelligent Customer Service: Combines text and image information to provide users with more accurate customer service support, such as handling product inquiries and troubleshooting.
  • Content Creation: Generates high-quality image-text content, such as image descriptions and creative copy, suitable for advertising, media, and creative industries.
  • Education: Assists teaching by helping students better understand and learn knowledge through a combination of images and text, for example explaining complex scientific charts.