LongCat-Next is an open-source discrete native multimodal model from Meituan's LongCat team, aiming to unify text, vision, and audio processing under the same autoregressive objective, minimizing inductive biases beyond the language paradigm. It is described as a foundation model with industrial-grade strength, at A3B scale, skilled at "seeing, creating, and speaking," and achieving strong performance on multiple multimodal benchmarks.
Purpose and Positioning This work primarily targets the fundamental obstacles of native multimodality, adopting a design philosophy that prioritizes simplicity and treats vision and audio as intrinsic extensions of language. The model is built on the LongCat-Flash-Lite MoE (A3B) backbone, serving as a multi-task learner that unifies language, vision, and audio within a single discrete framework. The official goal is to advance further research and development in the native multimodal direction within the community through open-source models and tokenizers.
Main Capabilities
- Unified Paradigm DiNA: Extends next-token prediction from language to native multimodality, internalizing diverse modalities into a shared token space, and simplifies multimodal modeling through modality-aware tokenizer-detokenizer pairs and by reusing the existing training infrastructure of large language models.
- Improved Discrete Visual Modeling: Combines the Semantic Alignment Encoder (SAE) with Residual Vector Quantization (RVQ) to generate hierarchical discrete tokens that preserve both semantic abstraction and fine-grained visual details, breaking through traditional representation limitations.
- dNaViT: Similar to a linguistic tokenizer, it proposes a highly flexible, unified visual discrete interface that extracts semantic features into "visual words," constructs a hierarchical representation space supporting dynamic tokenization and detokenization, and can seamlessly integrate with large language models without degrading performance.
- Unified Understanding and Generation: Under the DiNA framework, visual understanding and visual generation are reframed as two manifestations of the same prediction process, introducing minimal interference between them while preserving core language capabilities. The official claims it maintains strong generation quality even at a 28x compression ratio, especially in text rendering, and performs excellently in advanced speech understanding, low-latency speech dialogue, and customizable voice cloning.
Usage The official provides usage based on transformers, along with basic usage examples. Using LongCat-Next requires at least 3 GPUs (each with 80GB of VRAM, such as H100/A100 80GB), with a recommended environment of transformers >= 4.57.6 and accelerate >= 1.10.0. Before use, you need to modify WEIGHT_PATH_TO_LONGCAT_NEXT in ./config.json, as decompThe decoder uses lazy loading; see parse_model_response.py for detailed implementation and examples. The official documentation also provides a set of recommended sampling parameters, differentiated by task:
- Text: {"max_new_tokens":2048,"do_sample":false}
- Image understanding: {"max_new_tokens":1024,"do_sample":true,"temperature":0.4,"top_k":40,"top_p":0.85,"repetition_penalty":1.1}
- Image generation: {"max_new_tokens":2048,"do_sample":false,"visual_generation_config":{"do_sample":true,"temperature":0.5,"top_p":0.75,"top_k":1024,"custom_params":{"cfg_scale":3,"token_h":37,"token_w":37,"anyres_prefix":"<longcat_img_token_size>{h} {w}</longcat_img_token_size>"}}}
- Audio-to-text: {"max_new_tokens":1024,"do_sample":true,"temperature":0.2,"top_k":20,"top_p":0.85,"repetition_penalty":1.1}
- Audio-to-audio/speech synthesis: {"max_new_tokens":2048,"do_sample":true,"temperature":0.2,"top_k":20,"top_p":0.85,"repetition_penalty":1.1,"audio_generation_config":{"audio_parallel_decoding":false,"do_sample":true,"temperature":0.5,"top_k":5,"top_p":0.85,"repetition_penalty":1.3,"custom_params":{"sampling_rate":24000,"wave_concat_overlap":1200}}} Note that different inference frameworks support sampling parameters differently; in transformers, the inference parameter configuration is located at ./generation_config.json.
Deployment and Fine-Tuning Basic adaptation has been implemented in SGLang to support LongCat-Next deployment; see the meituan-longcat/LongCat-Next-inference repository for details. The official documentation also provides supervised fine-tuning (SFT) code based on PyTorch FSDP2, supporting understanding tasks (image+text→text), generation tasks (text→image), and unified mixed-sample training. For specific environment setup, data preparation, and example training scripts, see sft/ and its README.
License and Usage Notes This repository (including model weights and source code) is released under the MIT License, and any contribution to the repository is also licensed under the MIT License by default unless otherwise stated. This license does not grant any rights to use Meituan trademarks or patents; see the LICENSE file for details. The model has not been specifically designed or comprehensively evaluated for all possible downstream applications. Developers should consider the known limitations of large language models, including performance differences across languages, and evaluate accuracy, safety, and fairness before deploying in sensitive or high-risk scenarios; developers and downstream users must independently understand and comply with applicable laws and regulations, including but not limited to data protection, privacy, and content safety requirements. Nothing in the model card should be interpreted as changing or limiting the terms of the MIT License. When using it, you may cite its work; if you have questions, you can contact longcat-team@meituan.com or submit an issue.
Limitations The official materials do not provide pricing, fees, a specific list of supported platforms, detailed values for each benchmark, or complete deployment step details; such related content is not currently provided. Use requires a multi-GPU environment that meets the VRAM requirements.
