What is SPDL
SPDL (Scalable and Performant Data Loading) is an open-source data loading tool launched by Meta AI , capable of improving AI model training efficiency. Based on multithreading technology, it achieves high-throughput data loading and reduces computational resource consumption. Compared with traditional process-based methods, SPDL improves throughput by 2-3 times. It is compatible with Free-Threaded Python and can further improve performance by 30% in environments where the GIL is disabled. SPDL optimizes data loading based on an asynchronous event loop and thread-safe operations, supports distributed systems and the mainstream AI framework PyTorch, and is suitable for large-scale datasets and complex tasks.
Main features of SPDL
- High-throughput data loading: Achieves high throughput based on multithreading technology, meets the needs of high-speed GPU computing, and reduces GPU idle time.
- Low resource usage: SPDL uses fewer computing resources and reduces memory and CPU usage.
- Framework agnosticism: As a framework-agnostic data loading solution, SPDL can be used with different AI frameworks, including PyTorch.
- Compatibility: SPDL is compatible with Free-Threaded Python and can further improve performance in environments where the GIL is disabled.
- Performance monitoring and tuning: Provides tools to help users gain an in-depth understanding of the data loading process and perform performance optimization.
- Distributed system support: SPDL supports working in distributed systems. Whether on a single GPU or a large cluster, it can efficiently handle complex tasks.
- Prefetching and caching technology: Based on prefetching and caching technology, it ensures that the GPU always has data available for processing and reduces GPU idle time.
Technical principles of SPDL
- Thread-based parallelism: Thread-based parallel processing reduces the overhead of inter-process communication and improves data transmission speed.
- Asynchronous event loop: The core of SPDL is the asynchronous event loop, which is responsible for scheduling new tasks and responding to task completion, achieving true concurrent execution.
- Thread safety and GIL release: The media processing operations in SPDL are thread-safe and release the GIL (Python's Global Interpreter Lock) during execution, supporting true parallel execution.
- Pipeline abstraction: SPDL includes a task executor and provides a pipeline abstraction, allowing users to build complex data processing pipelines.
- Flexible concurrency adjustment: Users can adjust concurrency strategies separately according to different stages of data loading (such as data acquisition, preprocessing, and transmission) to optimize overall performance.
- Efficient media processing: Implements media decoding functionality from scratch, ensuring thread safety in performance-critical code and releasing the GIL.
- Asynchronous I/O operations: Uses the asynchronous APIs provided in network storage to perform asynchronous I/O operations, improving performance without being limited by the GIL.
SPDL project address
- Project official website：ai.meta.com/blog/spdl
- GitHub repository：https://github.com/facebookresearch/spdl
Application scenarios of SPDL
- Large-scale machine learning training: When training large-scale machine learning models, SPDL provides high-throughput data loading to ensure that GPU resources are fully utilized.
- Deep learning model training: Deep learning models benefit from SPDL's efficient media processing and data loading.
- Distributed training environments: In distributed training environments, work across multiple GPUs and nodes to provide consistent high-performance data loading.
- Real-time data processing: For applications that require real-time data processing, such as online recommendation systems or real-time monitoring systems, ensure fast data loading and processing.
- Multimodal data training: Multimodal AI model training involving multiple data types such as images, text, and audio benefits from the flexibility and efficiency of SPDL.