InstructCV

InstructCV is an official codebase presented at ICLR 2024 that trains text-to-image diffusion models to perform a wide variety of computer vision tasks via natural language instructions. Instead of building separate models for segmentation, depth estimation, or object detection, InstructCV unifies them under a single instruction-following paradigm. The project builds on pretrained diffusion models and fine-tunes them on a curated dataset of image-instruction-output triples, enabling zero-shot generalization across tasks. The primary audience is computer vision researchers and practitioners interested in multi-task learning, instruction-following models, and vision-language integration. The codebase provides training scripts, evaluation benchmarks, and pretrained checkpoints, making it accessible for academic experimentation and applied use. InstructCV works by conditioning diffusion models on both a text prompt (instruction) and an input image. During inference, users provide an image and a natural language command (e.g., "segment all cars in this image") and the model generates the corresponding output. This approach eliminates the need for task-specific architectures and enables rapid prototyping. What sets InstructCV apart is its ability to handle diverse vision tasks without task-specific heads or outputs, all while leveraging the generative power of diffusion models. The ICLR 2024 publication and open-source release underscore its novelty and reproducibility. Compared to dedicated task-specific models, InstructCV offers flexibility but requires careful tuning for production use.