What Is Profiling Data
Profiling Data is the performance analysis data of DeepSeek's open-source training and inference framework. Based on the detailed information captured during program execution by PyTorch Profiler, it is used to analyze and optimize software performance. After downloading, open it directly in the Chrome or Edge browser based on chrome://tracing or edge://tracing for visual analysis. Profiling Data records key metrics during program execution, such as time consumption, resource utilization, and communication patterns. In deep learning frameworks, Profiling Data helps developers understand the computation and communication overlap strategies during model training and inference, the usage of different hardware resources, and potential performance bottlenecks. Based on the analysis of the data, developers optimize code implementation, adjust parallelism strategies, and improve the overall efficiency of the system.
Main Features of Profiling Data
- Performance Bottleneck Identification: Based on recording the time consumption and resource usage during program execution, it helps developers quickly identify performance bottlenecks, such as which functions or modules consume too much time or resources.
- Resource Utilization Analysis: Analyzes the usage of hardware resources such as CPU, GPU, and memory, helping optimize resource allocation and avoid resource waste.
- Communication Pattern Analysis: In distributed systems, analyzes the communication patterns and latency between nodes, optimizes communication strategies, and reduces communication overhead.
- Optimization Guidance: Provides data support for code optimization, helping developers choose appropriate optimization strategies, such as parallelization and cache optimization.
Inference Process of Profiling Data
- Prefilling: Based on EP32 and TP1 (consistent with the actual online deployment of DeepSeek V3/R1), the prompt length is 4K, and the batch size per GPU is 16K tokens. Two micro-batches are used to overlap computation and all-to-all communication, ensuring balanced attention computation load between the two micro-batches.
- Decoding: Based on EP128 and TP1, the prompt length is 4K, and the batch size per GPU is 128 requests. Similar to prefilling, decoding also uses two micro-batches to overlap computation and all-to-all communication. Unlike prefilling, the all-to-all communication in the decoding phase does not occupy GPU SMs (Streaming Multiprocessors). After the RDMA (Remote Direct Memory Access) message is sent, all GPU SMs are released, and the system waits for the completion of the all-to-all communication after computation is completed.
Project Address of Profiling Data
- GitHub Repository：https://github.com/deepseek-ai/profile-data
Application Scenarios of Profiling Data
- Identifying Performance Bottlenecks: Quickly find the most time-consuming or resource-consuming parts of a program.
- Guiding Performance Optimization: Adjust strategies based on data to improve computation and communication efficiency.
- Analyzing Resource Utilization: Evaluate the usage of hardware resources (such as CPU, GPU, and memory) to avoid waste.
- Optimizing System Architecture: Adjust the configuration and communication strategies of distributed systems based on runtime data.
- Detecting Performance Regression: Compare performance data across different versions to ensure optimization is effective and no performance degradation occurs.