SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

mHC - A New Neural Network Architecture Introduced by the DeepSeek Team

Author: AI小集 · Original source: AI工具集

What is mHC

mHC (Manifold-Constrained Hyper-Connections) is a new neural network architecture design method introduced by the DeepSeek team, addressing the stability issues of traditional Hyper-Connections (HC) architectures in large-scale training. mHC restores the identity mapping property of residual connections by projecting the residual connection space of HC onto a specific manifold, effectively avoiding the problem of gradient explosion or vanishing. mHC utilizes the Sinkhorn-Knopp algorithm to project the residual connection matrix onto the manifold composed of doubly stochastic matrices, ensuring that the mean of the signal remains unchanged during propagation while strictly regulating the signal norm. mHC combines efficient infrastructure optimizations, such as kernel fusion, selective recomputation, and communication overlap, to ensure efficient implementation in large-scale models. Experiments show that mHC outperforms baseline models and HC in training stability, convergence speed, and downstream task performance, with only a slight increase in time overhead in large-scale training. As a general framework, mHC provides new ideas for deep learning architecture design and is expected to drive the evolution of next-generation foundation architectures.

mHC

Main Features of mHC

  • Restoration of Identity Mapping Property: By projecting the residual connection space onto a specific manifold, mHC restores the identity mapping property of residual connections, effectively solving the signal instability problem caused by traditional hyper-connection architectures when scaling to multiple layers, and significantly improving the stability of large-scale training.
  • Manifold Constraints and Signal Regulation: Utilizing the doubly stochastic matrix manifold, mHC ensures that the signal maintains a constant mean during propagation and strictly regulates the signal norm, effectively avoiding the problem of gradient explosion or vanishing, and significantly enhancing the stability of signal propagation.
  • Efficient Infrastructure Optimization: Through techniques such as kernel fusion, selective recomputation, and communication overlap, mHC achieves efficient operation in large-scale models, introducing only minimal training overhead, ensuring efficiency and scalability in practical applications.
  • Improved Model Performance: Experiments show that mHC significantly outperforms baseline models and traditional hyper-connection architectures in multiple downstream tasks, especially in complex tasks such as BBH and DROP, enhancing the model's reasoning ability and overall performance.
  • Scalability and Flexibility: As a general framework, mHC is applicable to various model scales, provides new directions for future exploration of different manifold constraints, and is expected to drive the evolution of next-generation foundation architectures.

Technical Principles of mHC

  • Manifold Projection: mHC constrains the properties of the connection matrix by projecting the residual connection matrix onto a specific manifold space (such as the Birkhoff polytope of doubly stochastic matrices). This projection ensures that the signal remains stable during propagation while preserving the ability for information interaction.
  • Application of Doubly Stochastic Matrices: Doubly stochastic matrices have the property that the sum of each row and each column is 1, which keeps the mean of the signal unchanged during propagation, strictly regulates the signal norm, and effectively avoids the problem of gradient explosion or vanishing.
  • Sinkhorn-Knopp Algorithm: mHC utilizes the Sinkhorn-Knopp algorithm to achieve manifold projection of the residual connection matrix. By iteratively normalizing rows and columns, any non-negative matrix is adjusted to a doubly stochastic matrix, achieving stable signal propagation.
  • Restoration of Identity Mapping: Through manifold constraints, mHC restores the identity mapping property of residual connections, ensuring stable signal propagation across multi-layer networks, and solving the training instability caused by the lack of identity mapping in traditional hyper-connection architectures.
  • Efficient Computation and Optimization: mHC combines techniques such as kernel fusion, mixed-precision computation, and selective recomputation to optimize computational efficiency and reduce memory access overhead, making this architecture highly efficient and scalable in large-scale model training.
  • Stability of Signal Propagation: By constraining the residual connection matrix, mHC ensures that signals remain stable during forward propagation and backward propagation, significantly reducing extreme values of signal gain and improving the stability and convergence speed of model training.

mHC Project Address

  • arXiv Technical Paper：https://arxiv.org/pdf/2512.24880

Application Scenarios of mHC

  • Large-Scale Language Model Pretraining: mHC can effectively improve the training stability of language models on large-scale datasets, and is suitable for pretraining tasks of ultra-large-scale language models such as 27B, significantly improving the model's convergence speed and performance.
  • Multi-Task Learning and Reasoning: In scenarios involving multiple downstream tasks (such as BBH, DROP, GSM8K, etc.), mHC helps models achieve better results in complex tasks by enhancing the model's reasoning ability and stability, improving the efficiency of multi-task learning.
  • Efficient Distributed Training: mHC combines optimized infrastructure design, such as kernel fusion and communication overlap, and is suitable for distributed training environments, especially in large-scale clusters where it can significantly reduce training overhead and improve training efficiency.
  • Evolution of Next-Generation Architectures: As a general framework, mHC provides a foundation for exploring novel neural network architectures, supports researchers in innovating under different manifold constraints and optimization strategies, and promotes the evolution of next-generation infrastructure.
  • Resource-Constrained Environments: Because mHC introduces minimal computational overhead while maintaining performance, it is suitable for resource-constrained environments such as mobile devices or edge computing scenarios, enabling efficient model deployment under limited computational resources.
  • Academic Research and Theoretical Exploration: mHC provides a new perspective for studying signal propagation and architecture design in deep learning, and is suitable for in-depth exploration of neural network topologies and optimization strategies in academic research.