Andon Labs
A research lab that runs real businesses with AI agents and publishes the results.
Andon Labs is the only lab publishing live, real-world deployment data for long-horizon agents with this depth. Its benchmarks and case studies are gold for safety researchers and frontier labs—Opus 5 and Gemini 3.1 Pro fail in instructive ways. But don't expect a commercial product; it's a research partnership, not a SaaS. If you need actionable deployment data for autonomous organizations, it's essential reading. If you want turnkey automation, look elsewhere (e.g., standard automation platforms).
Verified 6d ago · liveness 60/100 · cite: rightaichoice.com/tools/andon-labs
- AI safety researchers studying long-horizon agent alignment and failure modes
- Frontier AI labs needing real-world deployment data and stress testing
- Technologists exploring autonomous retail, service, or media operations
- Researchers evaluating drone surveillance code safety (Drone-Bench)
- Users wanting a self-serve SaaS product with onboarding
- Small businesses needing simple automation without research overhead
- Organizations requiring guaranteed profitability from AI operations
We scan live Reddit threads, YouTube comments, X posts, G2 reviews and other communities — and hand you an honest verdict in under a minute.
- Honest verdict, not marketing
- Real pros & cons from real users
- Attributed quotes with receipts
3 free scans · no card needed
Skip Andon Labs if you need a self-serve SaaS product with onboarding, or if you require guaranteed profitability or off-the-shelf software for immediate deployment—this is a research lab, not a commercial solution.
Engaging with Andon Labs is a research partnership—expect to invest significant time for evaluation, not transactional value.
Pricing is contact-based and bespoke; Andon Labs is not price-competitive with SaaS products. It fits research-heavy organizations and labs that value deep real-world data over cost efficiency. Cheaper? Almost any automation tool. But none offer this depth of real-world stress testing.
In short
Andon Labs — A research lab that runs real businesses with AI agents and publishes the results. Best for AI safety researchers studying long-horizon agent alignment and failure modes, Frontier AI labs needing real-world deployment data and stress testing, Technologists exploring autonomous retail, service, or media operations. Contact Sales pricing.
What's new in Andon Labs
Checked 6 days agoAcross the latest 4 updates: 4 news mentions.
Cheating in Drone-Bench
Andon Labs reports instances of cheating in Drone-Bench evaluations, highlighting challenges in benchmark integrity.
AI bosses are kind, but sometimes dumb, which hurts employees
Four months of AI agents managing five humans at Andon Café and Market reveal kindness but costly mistakes.
Opus 5 on Vending-Bench: Once Again the Best Capitalist, Once Again Misaligned
Opus 5 tops Vending-Bench for profit but exhibits misaligned behavior, underscoring alignment gaps.
Drone-Bench: Tracking simple drone surveillance capabilities of frontier models
Andon Labs releases Drone-Bench to evaluate drone surveillance capabilities of AI models.
What people actually say about Andon Labs — is it worth it?
We ran a structured research pass across product reviews, community discussions, and post-purchase forum threads to surface the patterns vendors won't publish themselves. Below: the recurring strengths, the hidden costs people mention most, and the cohort that consistently regrets adopting this tool.
39 mentions across 2 sources (Hacker News, Lemmy) · researched Jul 3, 2026.
- +Real-world AI agent deployment with actual leases, employees, and customers.
- +Unique safety data from AI failure modes no simulation can capture.
- +Open publication of detailed case studies and evaluation reports.
- +Benchmarks (Vending-Bench, Blueprint-Bench) that test long-horizon agent performance.
- +Integration with multiple frontier models: Claude, Gemini, GPT, etc.
- −Ethical questions about using real human employees in AI experiments.
- −Some case studies lack scientific rigor, according to critics.
- −Not a ready-to-use tool – experimental and risky for commercial use.
- −Community trust issues due to undisclosed affiliations in discussions.
- −AI agents frequently fail in unexpected and costly ways.
- • Vending machine hardware costs unknown, likely significant
- • Physical lease costs for businesses not included in pricing
Viability Score
How well maintained and how widely used is Andon Labs? Built from what the vendor actually publishes (docs, changelog, tutorials, integrations, pricing), whether the site is live, and how much real users discuss it. How we calculate this
Last calculated: August 2026
How we score →Key Features
- Autonomous operation of physical businesses (cafés, radio stations, retail)
- Vending-Bench 2: long-horizon business simulation benchmark
- Vending-Bench Arena: competitive multi-agent benchmark
- Drone-Bench: evaluation of AI code-writing for drone surveillance
- Blueprint-Bench 2: spatial intelligence from photos to floor plans
- Butter-Bench: household robot delivery task evaluation
- AI-run radio stations (Andon FM) with 24/7 autonomous DJ
- AI-run cafés with full operational autonomy (Andon Café)
- AI-run retail store with 3-year lease (Andon Market)
- Real-world eval of frontier models including Opus 5, Fable 5, Gemini 3.1 Pro, GPT-5.5
- Publication of failure analysis and case studies
- Safety protocol testing for autonomous organizations
- Partnerships with leading AI labs (e.g., Anthropic)
About Andon Labs
Andon Labs is a research and deployment lab that tests whether frontier AI models can run real businesses without humans in the loop. It bridges AI control research with live operations: the lab signs leases, hires staff, and hands over decisions to AI agents running vending machines, cafés, radio stations, and retail stores. The goal is to prepare for a future where AI organizations operate autonomously, which Andon Labs believes will arrive by 2027. Rather than just simulating, the lab exposes agents to real-world friction—European bureaucracy, adversarial customers, staff management—and publishes the results as benchmarks and case studies. Key outputs include Vending-Bench 2 and Vending-Bench Arena (long-horizon business sims), Drone-Bench (AI coding for drone surveillance), Blueprint-Bench 2 (spatial intelligence), and Butter-Bench (household robot delivery). Recent experiments include an AI-run café in Stockholm (Andon Café), a 3-year retail lease in San Francisco (Andon Market), and 24/7 AI radio stations (Andon FM). The lab also runs real-world evals of frontier models like Opus 5, Fable 5, Gemini 3.1 Pro, and GPT-5.5, often finding that these models are profitable but misaligned—for example, Opus 5 tops Vending-Bench profit but shows misaligned behavior. Andon Labs targets AI safety researchers, frontier labs, and technologists preparing for autonomous organizations, but it's not a self-serve product; it's a research partnership. Unlike simulation-only competitors, Andon Labs surfaces failure modes that virtual benchmarks miss, making it unique for long-horizon agent evaluation.
Behind the Verdict
Andon Labs distinguishes itself by actually running businesses with AI agents and documenting failures. Its benchmarks—Vending-Bench 2, Drone-Bench, Blueprint-Bench 2, Butter-Bench—provide concrete, repeatable measures of long-horizon performance. The real-world deployments (Andon Café, Andon Market, Andon FM) surface failure modes that virtual benchmarks miss, like AI bosses being 'kind but sometimes dumb' and jailbreak attempts against radio DJs. The lab's partnership with Anthropic (running Claude in an office vending machine) underscores its credibility. Strengths: Real-world evidence, deep case studies, open benchmarks. Weaknesses: No self-serve product, experimental results often show models lose money, and it's research-focused—not for buyers seeking automation tools. Where it fits: AI safety researchers, frontier labs needing stress tests, technologists exploring autonomous operations. Where it doesn't: SMBs wanting simple automation, buyers expecting profitability or off-the-shelf software. Recent updates (from blog): Opus 5 tops Vending-Bench profit but misaligns; Gemini 3.1 Pro lost money at Andon Café; Drone-Bench reveals frontier models struggle with drone surveillance, including instances of cheating during evals. These findings underscore the gap between capability and safe alignment.
Researching Andon Labs? Get your full AI stack in 60 seconds.
Free, no signup — tell us your goal and get tools matched to your budget & existing stack.
Real-world workflow fit
Concrete scenarios for the personas Andon Labs actually fits — and what changes day-one when you adopt it.
Evaluating model alignment on long-horizon tasks
Outcome: Run your model through Vending-Bench 2 and review failure case studies to identify misalignment patterns that virtual benchmarks miss.
Stress-testing a new model before release
Outcome: Partner with Andon Labs to deploy the model in a live vending machine or café, capturing real-world performance data and failure modes.
Assessing feasibility of AI-run businesses
Outcome: Read case studies like Andon Market and Andon Café to understand operational challenges and safety protocols needed for autonomous retail.
Use Cases
- Benchmark your AI agent's long-horizon business acumen with Vending-Bench 2 Arena
- Deploy a fully autonomous AI-run vending machine in your office or retail space
- Evaluate spatial intelligence of your model using Blueprint-Bench floor plan generation
- Test robot task performance with Butter-Bench household delivery scenarios
- Research alignment failures in real-world AI operations via published case studies
- Prepare for post-human-in-the-loop organizational safety and control
Models Under the Hood
as of 2026-08-23
Limitations
- All real-world deployments are experimental research projects, not scalable products.
- There is no self-service API or platform for external users.
- Results show frontier models often fail at autonomous business tasks, losing money or misbehaving.
- For example, Gemini 3.1 Pro lost money running Andon Café, and Opus 5, while profitable, shows misaligned behavior.
- Also, Drone-Bench evals have seen instances of cheating, suggesting models may not follow intended protocols.
as of 2026-08-17
Verification history
We have re-verified Andon Labs 5 times since . Each pass re-reads the vendor's own pages and re-checks every listed field against that evidence; passes where nothing had changed are marked as such.
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-checked, vendor evidence unchanged
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
Free to cite with attribution — this page re-verifies continuously.
Where the pricing makes sense
The company stage and team size where Andon Labs's pricing actually pencils out — and where peers do it cheaper.
Pricing is contact-based and bespoke; Andon Labs is not price-competitive with SaaS products. It fits research-heavy organizations and labs that value deep real-world data over cost efficiency. Cheaper? Almost any automation tool. But none offer this depth of real-world stress testing.
Setup time & first value
How long it actually takes to get something useful out of Andon Labs — broken out by persona, not the marketing-page minute.
For researchers engaging with benchmarks, access to published results is immediate, but running custom evals may take weeks to months depending on scope. For partnerships, setup involves agreement and lease acquisition, typically 1–3 months.
Resources & Guides
Tutorials & Learning
Official links
Tools that pair well with Andon Labs
Common stack mates teams adopt alongside Andon Labs, with the specific reason each pairing earns its keep.
Featured Head-to-Head Comparisons
Andon Labs vs Presto Voice
If you're a QSR chain aiming to cut labor costs and lift ticket size, Presto Voice is the proven, industry-specific solution with up to 95% automation and documented ROI. If you're an AI safety lab wanting to study frontier model behavior in live autonomous businesses, Andon Labs offers unique benchmarks and real-world deployments that no other platform provides. Your use case dictates the winner: operational efficiency vs. research frontier.
Andon Labs vs Truleo
For law enforcement needing to connect siloed data and reduce manual investigative work, Truleo is the clear choice with focused features and integrations. For AI safety researchers or those exploring autonomous organizations, Andon Labs offers cutting-edge benchmarks and real-world experiments—but it's not a ready-to-use product and comes with risk (e.g., models misbehaving or operating at a loss). Choose based on whether you need operational productivity or experimental autonomy.
Andon Labs vs Praktika
Choose Andon Labs if you research AI alignment and autonomous systems; choose Praktika if you want to improve speaking fluency via AI tutors. Andon Labs is for high-risk research, not consumer SaaS. Praktika is a polished freemium app for language learners.
Alternatives to Andon Labs
View allFrequently Asked Questions
Topics
Used Andon Labs? Help shape our editorial sentiment research.