MagicDrive

MagicDrive is an open-source research framework from CUHK, HKUST, and Huawei Noah's Ark Lab that generates photorealistic street-view images and videos with fine-grained 3D geometry control. Built for autonomous driving researchers and engineers, it enables controllable synthetic data for training perception models, especially BEV segmentation and 3D object detection. Unlike generic text-to-image models, MagicDrive lets you specify camera poses, BEV road maps, 3D bounding boxes, and text descriptions to synthesize scenes with accurate object dimensions, occlusion, and road elevation. The framework uses customized encoding strategies for each control type and introduces a cross-view attention module to maintain multi-camera consistency. It supports both UNet and DiT backbone architectures. As of December 2024, all pretrained checkpoints are publicly available—three image-generation models (6x224x400, 6x272x736, 6x424x800) and two video-generation models (16-frame, 61-frame)—so you can start generating without training. MagicDrive also offers diverse generation: sample unlimited street views from the same geometric conditions via Slerp latent interpolation, produce continuous scene sequences from annotation sequences, and edit objects individually—moving, removing, or changing attributes while keeping everything else fixed. The video extension, fine-tuned with Tune-a-Video on nuScenes, produces temporally consistent clips, including up to 60 frames at 12 fps (5 seconds). This is a research tool, not a production API. It requires expertise in diffusion models and 3D geometry pipelines, and generation is slow for real-time use. But for AV perception teams needing precise, controllable synthetic data, MagicDrive offers a level of 3D control that general-purpose generators can't match. It's positioned as a baseline in workshops like W-CODA at ECCV24 and presented at Huawei's HDC2024.