SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

TeleChat3 - China Telecom's trillion-parameter MoE semantic large model

Author: AI小集 · Original source: AI工具集

What is TeleChat3

TeleChat3 is a trillion-parameter fine-grained MoE semantic large model independently developed by China Telecom Artificial Intelligence Research Institute (TeleAI). It is trained on domestic computing power and supports tasks such as multi-turn dialogue, text creation, and code generation. The model contains 105B parameters (with 4.7B actually activated) and achieves precise knowledge invocation through 192 routed experts. It excels at handling complex tasks such as game design and web development. Its distinguishing feature lies in full-chain domestic technology, with training efficiency reaching 93%. It has been applied to scenarios such as intelligent customer service and government Q&A, and a 36B dense model version has been open-sourced.

TeleChat3

Main features of TeleChat3

  • Code generation and optimization: It can efficiently break down task requirements, integrate multiple code capabilities, and deliver complete runnable code in one go, enabling many types of tasks such as code completion, code modification, and website and web game generation, saving a large amount of manual debugging time, running smoothly, and producing accurate results.
  • Knowledge Q&A and logical reasoning: It has achieved capability improvements in dimensions such as knowledge Q&A and logical reasoning, can better understand and answer users' questions, and provide accurate knowledge information and logical reasoning results.
  • Text creation and semantic understanding: It has text creation capabilities and can generate high-quality text content. It has also undergone targeted optimization in semantic understanding, allowing it to better understand the text information input by users.
  • Role-playing and multi-turn dialogue: It supports role-playing functionality and can conduct dialogues according to roles and scenarios set by users. In multi-turn dialogues, it can automatically process historical information and maintain the coherence and consistency of the conversation.
  • Introduction of "Thinking mode": By adding specific guiding symbols to the dialogue template, the model can automatically generate intermediate reasoning steps, significantly improving logic and accuracy in complex tasks and making the AI reasoning process traceable.

Technical principles of TeleChat3

  • Model architecture: TeleChat3 adopts a fine-grained Mixture-of-Experts (MoE) architecture. For example, the TeleChat3-105B-A4.7-Thinking model contains 1 shared expert and 192 routed experts, activating 4 experts each time.
  • Reasoning mechanism: TeleChat3 supports long-text reasoning optimization. Through NTK-aware extrapolation and attention scaling methods, it can extrapolate a model fine-tuned on a shorter length to a longer length during inference.
  • Training and optimization: The series of models is trained entirely on domestic computing power (such as the Shanghai Lingang domestic ten-thousand-card computing power pool), with a basic data volume of 15T tokens. At the same time, the model has achieved deep adaptation to Ascend Atlas 800T A2 training servers and the MindSpore framework.
  • Thinking mode: The entire TeleChat3 series supports Thinking mode, generating the reasoning process by adding specific guiding symbols to the chat template, thereby improving logic and accuracy in complex tasks.
  • Positional encoding and attention mechanism: It uses Rotary Embedding to inject positional information into tokens. Through rotation operations on queries (Q) and keys (K), the model captures the sequential dependency relationships of the sequence. At the same time, it supports a causal self-attention mechanism to ensure that the prediction at each position depends only on the preceding tokens.
  • Nonlinear enhancement: The model includes a multilayer perceptron (MLP) module, which is used to perform nonlinear transformation on the output of the attention layer, enhancing the model's expressive ability.

TeleChat3 project address

  • GitHub repository：https://github.com/Tele-AI/TeleChat3
  • ModelScope：
    • https://www.modelscope.cn/models/TeleAI/TeleChat3-105B-A4.7B-Thinking
    • https://www.modelscope.cn/models/TeleAI/TeleChat3-36B-Thinking

Application scenarios of TeleChat3

  • Code generation and development assistance: It can be used for code completion, code modification, and generating websites and web games, etc., helping developers quickly complete programming tasks and improve development efficiency.
  • Knowledge Q&A and education: It performs well in tasks such as knowledge Q&A and logical reasoning, and can provide users with accurate knowledge information. It is suitable for the education field, assisting learning and answering questions.
  • Text creation and content generation: It can generate high-quality text content and can be used in scenarios such as writing, copywriting, and news reporting, meeting different text creation needs.
  • Multi-turn dialogue and customer service: It supports multi-turn dialogue, can automatically process historical information, and maintains conversational coherence. It is suitable for scenarios such as intelligent customer service and virtual assistants.
  • Role-playing and interactive entertainment: It supports role-playing functionality and can conduct dialogues according to user-set roles and scenarios, and is applied in fields such as games and interactive entertainment.