Calvin
Open-source benchmark for long-horizon language-conditioned robot manipulation.
CALVIN remains a definitive benchmark for long-horizon language-conditioned manipulation, with a clear leaderboard and well-defined splits. It is indispensable for researchers pushing toward multi-task, few-shot generalization but lacks real-world deployment capabilities; it is strictly a simulation evaluation tool.
- Robotics researchers studying language-conditioned manipulation
- ML researchers developing long-horizon multi-task reinforcement learning
- Researchers in vision-language models for embodied AI
- PhD students or academic labs benchmarking manipulation policies
- Production robot deployment or real-world applications out-of-the-box
- Beginners without strong RL or robot simulation background
- Users looking for a full API or pre-trained models; it's a benchmark environment only
We scan live Reddit threads, YouTube comments, X posts, G2 reviews and other communities — and hand you an honest verdict in under a minute.
- Honest verdict, not marketing
- Real pros & cons from real users
- Attributed quotes with receipts
3 free scans · no card needed
In short
Calvin — Open-source benchmark for long-horizon language-conditioned robot manipulation. Best for Robotics researchers studying language-conditioned manipulation, ML researchers developing long-horizon multi-task reinforcement learning, Researchers in vision-language models for embodied AI. Free to use.
Viability Score
How likely is Calvin to still be operational in 12 months? Based on 4 signals — momentum (how recently it shipped), wrapper dependency, revenue model, and web presence.
Last calculated: July 2026
How we score →Key Features
- Open-source simulated benchmark for language-conditioned manipulation
- Long-horizon tasks with up to 5 instructions in a row
- Four distinct environments (A, B, C, D) for cross-scene generalization
- Supports static RGB, gripper RGB, depth, and tactile sensor suites
- Predefined task sequences with natural language annotations
- Leaderboard tracking policy performance across standard splits
- Metrics: MTLC (multi-task classification) and LH-MTLC (long-horizon)
- Integration with PyBullet physics simulator
- Baseline implementations for multiple input modalities
- Published train/test splits for reproducible research
- Evaluates compositional skills like 'push red block' then 'open drawer'
- Open-source code and data on GitHub under MIT license
- Includes cross-scene generalization evaluation (A,B,C→D)
About Calvin
CALVIN (Composing Actions from Language and Vision) is an open-source simulated benchmark for evaluating language-conditioned policy learning in long-horizon robot manipulation tasks. Designed for robotics and AI researchers, it provides standardized environments where agents must execute sequences of up to five manipulation tasks specified via natural language, using onboard sensors like static RGB, gripper camera, depth, and tactile inputs. The benchmark includes four distinct environments (A, B, C, D) for cross-scene generalization, with a leaderboard tracking state-of-the-art results. Key features include compositional language instructions, long-horizon tasks without reset, and metrics like MTLC and LH-MTLC. Unlike commercial simulators, CALVIN is purely a research benchmark, free and open-source under MIT license, with baseline implementations and PyBullet integration. It enables reproducible comparison across methods and drives progress in multi-task, few-shot generalization for embodied AI.
Behind the Verdict
CALVIN is the standard evaluation suite for language-conditioned robot manipulation research. Its strength lies in forcing agents to chain multiple skills from raw language instructions, mimicking real-world complexity. The leaderboard, especially the A,B,C→D split, pinpoints generalization gaps. We'd reach for it when benchmarking a new policy or comparing against recent methods like FLOWER or MDT. Where it bites: it's simulation-only, no real-robot transfer wrapper, and the tasks, while diverse, remain scripted. The closest alternative is MetaWorld or RLBench, but neither focuses on language-guided long-horizon chaining as explicitly. For production robotics, look elsewhere; for publishable RL evaluation, it's essential.
Researching Calvin? Get your full AI stack in 60 seconds.
Free, no signup — tell us your goal and get tools matched to your budget & existing stack.
Use Cases
- Evaluate a new vision-language policy on standardized long-horizon manipulation tasks.
- Compare multi-task learning approach against baselines across environment variants.
- Train agents to chain multiple language instructions without resetting.
- Assess generalization by training on three environments and testing on the fourth.
Limitations
- CALVIN is strictly a simulation benchmark; results may not transfer directly to real robots.
- It does not provide pre-trained agents, APIs, or support; users must implement their own policies.
- The benchmark's tasks are limited to the provided environment and do not cover open-world manipulation.
Tools that pair well with Calvin
Common stack mates teams adopt alongside Calvin, with the specific reason each pairing earns its keep.
Featured Head-to-Head Comparisons
Alternatives to Calvin
View allFrequently Asked Questions
Categories
Best-of guides
Topics
Used Calvin? Help shape our editorial sentiment research.