SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

NEO - SenseTime and Nanyang Technological University Open-Source a Brand-New Multimodal Model Architecture

Author: AI小集 · Original source: AI工具集

What is NEO

NEO is a brand-new multimodal model architecture jointly developed by SenseTime and Nanyang Technological University. As the first native multimodal architecture (Native VLM), NEO breaks through the limitations of traditional multimodal models through deep innovation at the underlying architecture level. Core innovations include Native Patch Embedding, which can capture image details more finely; Native 3D Rotary Position Embedding (Native-RoPE), which perfectly adapts to the natural structure of images and text; and Native Multi-Head Attention, which enhances the model's ability to understand complex image-text relationships. NEO performs excellently in data efficiency, performance, and inference cost-effectiveness. It requires only a small amount of data to achieve top-tier visual perception capabilities and has achieved excellent results in multiple authoritative evaluations. SenseTime has open-sourced NEO models in 2B and 9B specifications, promoting the industrialization of native multimodal technology and building the next-generation multimodal technology standard.

NEO

NEO's Main Features

  • Native Multimodal Fusion: Through its underlying architecture design, NEO achieves deep fusion of images and text, breaking through the modular limitations of traditional multimodal models and handling mixed image-text content more naturally.
  • Efficient Data Utilization: With only a small amount of data (such as 390 million image-text examples), NEO achieves top-tier visual perception capabilities, significantly improving data utilization efficiency and reducing training costs.
  • Outstanding Performance: In multiple authoritative evaluations, NEO demonstrates excellent performance. Whether for image understanding, text generation, or image-text reasoning tasks, it can provide high-quality output.
  • High Inference Cost-Effectiveness: Especially at small and medium parameter scales (such as 0.6B-8B), NEO performs excellently in edge deployment and inference efficiency, making it suitable for a wide range of practical application scenarios.
  • Open-Source Collaboration and Expansion: SenseTime has open-sourced NEO models in 2B and 9B specifications, encouraging developers and researchers to carry out further development and applications based on this architecture, promoting the industrialization of multimodal technology.

NEO's Technical Principles

  • Native Patch Embedding: Through bottom-up continuous mapping, image pixels are directly embedded into the model, avoiding the discretization processing of traditional image tokenizers, capturing image details more finely, and improving image modeling capability.
  • Native 3D Rotary Position Embedding (Native-RoPE): Innovatively decouples the 3D spatiotemporal frequency allocation of images and text, assigning high-frequency encoding to images and low-frequency encoding to text, better adapting to the natural structure of the two modalities and supporting spatial structure associations in complex scenarios.
  • Native Multi-Head Attention: Within a unified framework, it simultaneously implements autoregressive attention for text and bidirectional attention for vision, enhancing the model's ability to understand image-text relationships and supporting complex mixed image-text understanding and reasoning tasks.
  • Underlying Architecture Innovation: Starting from the underlying architecture, it achieves deep multimodal fusion rather than simple modular concatenation, fundamentally breaking through the performance bottlenecks of traditional multimodal models and improving the model's overall effectiveness.
  • Efficient Training and Inference: Through optimized architecture design, NEO demonstrates higher efficiency in training and inference. Especially at small and medium parameter scales, it can achieve lower computational costs and higher inference speeds, making it suitable for a wide range of practical applications.

NEO's project address

  • Github repository：https://github.com/EvolvingLMMs-Lab/NEO
  • arXiv technical paper：https://arxiv.org/pdf/2510.14979

NEO's application scenarios

  • Image and text generation: NEO can generate high-quality images based on text descriptions, or generate accurate text descriptions based on image content, suitable for creative design, content creation, and other fields.
  • Intelligent search and recommendation: By understanding the deep semantics of images and text, NEO can provide users with more precise search results and personalized recommendations, enhancing the user experience.
  • Multimodal question answering: NEO can handle question-answering tasks involving mixed images and text, combining image and text information to provide more accurate answers, suitable for scenarios such as education and customer service.
  • Intelligent driving and robot vision: NEO's efficient image understanding capability can be used for scene perception and object recognition in intelligent driving, as well as environmental understanding and navigation in robot vision.
  • Industrial inspection and monitoring: NEO can quickly and accurately identify anomalies and defects in images, suitable for quality inspection and monitoring systems in industrial production.
  • Medical imaging analysis: NEO can assist doctors in the analysis and diagnosis of medical images, combining medical record text information to provide more comprehensive diagnostic recommendations.