Potato

Potato is an open-source annotation platform from the University of Michigan, built for research teams who want to define their labeling task in a single YAML file rather than through a vendor's form builder. You install it with pip install potato-annotation, launch it with potato start config.yaml, and start annotating at localhost:8000. No seat licenses, no per-label metering, and your data stays on your own infrastructure. The breadth is the draw. Potato packages 61 annotation types spanning text, image, audio, video, and 3D: radio buttons, multiselect, Likert scales, sliders, span labeling, best-worst scaling, pairwise comparison, bounding boxes, polygons, events, taxonomies, audio waveforms, video playback, point clouds, and depth maps. QDA Mode adds a living codebook, in-vivo codes, memos, and cases for qualitative coding work. The same install covers NLP classification, vision bounding boxes, speech transcription, and survey-style data collection. Agent evaluation is where it separates from the crowd. You can bring traces from any framework, label a multi-agent run on a clickable interaction graph, evaluate computer-use, voice, and video agents, calibrate an LLM judge against human labels, edit trajectories into SFT or DPO training data, and rank models in an arena — all from YAML. Five active-learning strategies (uncertainty sampling, diversity-based selection, BADGE, BALD, and a hybrid ensemble) plus an LLM cold start help you find the instances worth labeling before you have any labels at all. LLM assistance runs through OpenAI, Claude, and Gemini for hints, keyword highlighting, label suggestions, MACE competence estimation, and diversity ordering. Against Labelbox or Prodigy, Potato trades vendor-managed hosting and support contracts for zero license cost and unlimited annotators — provided you have someone who can run a Python server.