What is HoloDrive
HoloDrive is an overall 2D-3D multimodal street scene generation framework for autonomous driving proposed by SenseTime, Shanghai Artificial Intelligence Laboratory, and other institutions. The framework jointly generates camera images and LiDAR point clouds, filling the gap in 2D-3D multimodal joint generation in autonomous driving. HoloDrive adopts BEV-to-Camera and Camera-to-BEV conversion modules between heterogeneous generative models, and introduces a depth prediction branch into the 2D generative model to eliminate the projection ambiguity from image space to BEV space.
Main Features of HoloDrive
- Joint generation of camera images and LiDAR point clouds: HoloDrive can simultaneously generate multi-view camera images and LiDAR point clouds, filling the gap in 2D-3D multimodal joint generation in autonomous driving.
- Cross-modal structure: Through the BEV-to-Camera and Camera-to-BEV conversion modules, as well as the depth prediction branch in the 2D generative model, HoloDrive achieves effective alignment and information exchange between 2D and 3D spaces, enabling the entire model to be trained end-to-end.
- Temporal structure and progressive training: By incorporating temporal structure and carefully designed progressive training, HoloDrive extends the method to predict the future, making it suitable for both single-frame generation and video generation tasks.
- High-performance generation: Experimental results on the NuScenes dataset show that HoloDrive achieves optimal performance in generating single-frame and sequence data of multi-view camera images and LiDAR point clouds, significantly outperforming existing state-of-the-art methods (SOTA).
Technical Principles of HoloDrive
- BEV-to-Camera and Camera-to-BEV conversion modules: BEV-to-Camera and Camera-to-BEV conversion modules are adopted between heterogeneous generative models to align 3D and 2D spaces.
- Depth prediction branch: A depth prediction branch is introduced into the 2D generative model, with natural supervision derived from 3D LiDAR, eliminating the projection ambiguity from image space to BEV space.
- Temporal structure: By incorporating temporal structure, HoloDrive can extend the method to predict the future, making it suitable for both single-frame generation and video generation tasks.
- Progressive training: Based on a progressive training strategy, combined with additional multi-task learning from the video domain, it achieves a smooth transition in the training phase.
HoloDrive Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2412.01407
Application Scenarios of HoloDrive
- Generating realistic street scenes: HoloDrive can jointly generate multi-view camera images and LiDAR point clouds to create realistic street scenes, reducing the need for expensive manual modeling of the real world.
- 2D-3D joint generation: Through the BEV-to-Camera and Camera-to-BEV conversion modules, as well as the depth prediction branch in the 2D generative model, HoloDrive achieves effective alignment and information exchange between 2D and 3D spaces.
- Temporal structure and progressive training: By incorporating temporal structure and carefully designed progressive training, HoloDrive extends the method to predict the future.