Promptfoo
Automated red teaming and LLM vulnerability scanning that runs inside your CI/CD pipeline
If your team already reviews code in pull requests, Promptfoo is the least disruptive way to get real adversarial testing into that loop. The free Community tier is a genuine product, not a teaser: all evaluation features, every model provider, and red teaming up to 10k probes per month, self-hosted. Watch two things before committing: Enterprise and On-Premise are custom quotes (you buy extra probes beyond the 10k cap), and OpenAI agreed to acquire the vendor in March 2026. Compare against Lakera if you mainly want runtime guardrails, and LangSmith or Braintrust if evaluation is the bigger need.
Verified 6h ago · liveness 89/100 · cite: rightaichoice.com/tools/promptfoo
- AppSec and security teams running automated red teaming across multiple LLM apps and agents
- Developers who want AI security findings surfaced in CI/CD runs and pull requests
- Teams building RAG systems and agents that need business-logic-aware vulnerability detection
- Financial services, insurance, telecom, and real estate teams with industry compliance needs
- Prototypes with no real users, where adversarial testing is premature
- Teams that want a runtime guardrail they can switch on without authoring tests
- Orgs with nobody comfortable editing YAML or wiring CI jobs
We scan live Reddit threads, YouTube comments, X posts, G2 reviews and other communities — and hand you an honest verdict in under a minute.
- Honest verdict, not marketing
- Real pros & cons from real users
- Attributed quotes with receipts
3 free scans · no card needed
Skip Promptfoo if you want a runtime guardrail you can switch on without writing YAML tests or wiring a CI job — this is a developer-workflow product, and the setup only pays off if someone on your team owns test authoring.
A probe is one request to your target, so the free tier's 10k-per-month allowance runs out fast once you scan continuously or across several applications — Enterprise is where you buy more.
Community is free forever and unusually complete — all evaluation features, every model provider, and red teaming up to 10k probes per month, self-hosted on your own infrastructure. That undercuts paid-per-seat evaluation tools like Braintrust or LangSmith for a solo developer or small team. The jump is steep: everything beyond basic scanning — advanced detection, the guardrail dashboard, SSO, webhooks, teams — is Enterprise, custom-quoted by team size and needs. Budget for a sales cycle, not a
In short
Promptfoo — Automated red teaming and LLM vulnerability scanning that runs inside your CI/CD pipeline. Best for AppSec and security teams running automated red teaming across multiple LLM apps and agents, Developers who want AI security findings surfaced in CI/CD runs and pull requests, Teams building RAG systems and agents that need business-logic-aware vulnerability detection. Free to use.
What's new in Promptfoo
Checked todayAcross the latest 1 update: 1 news mention.
What people actually say about Promptfoo — is it worth it?
We scanned public community sources for Promptfoo on Oct 14, 2026 and could not establish that the discussion we found is about this tool rather than something else sharing its name. Only 3 of the posts we fetched could be positively tied to Promptfoo. Rather than publish a sentiment score built on the wrong subject, we publish nothing here and re-run the scan.
Viability Score
How well maintained and how widely used is Promptfoo? Built from what the vendor actually publishes (docs, changelog, tutorials, integrations, pricing), whether the site is live, and how much real users discuss it. How we calculate this
Last calculated: October 2026
How we score →Key Features
- Automated red teaming for LLM apps, agents, and RAG pipelines
- Context-aware attack generation: direct and indirect prompt injections
- Jailbreaks tailored to your specific guardrails
- Data and PII leak detection
- Business rule violation testing
- Insecure tool-use testing in AI agents
- Toxic content generation testing
- Coverage of 50+ vulnerability types
- Real-time Guardrails against jailbreaks and adversarial attacks
- Model Security: testing and monitoring for AI models
- MCP Proxy for securing Model Context Protocol traffic
- Code Scanning for LLM vulnerabilities in VS Code and JetBrains IDEs
- LLM evaluations for prompts, models, and RAG pipelines
- Remediation guidance and issue tracking inside pull requests
- CI/CD integration with continuous monitoring and webhooks
About Promptfoo
Promptfoo is an AI security testing platform for engineering teams shipping LLM apps, agents, and RAG pipelines. You point it at your target and it generates application-specific attacks rather than canned jailbreaks: direct and indirect prompt injections, jailbreaks tailored to your guardrails, PII and data leaks, business rule violations, insecure tool use in agents, and toxic content generation. The vendor cites coverage of 50+ vulnerability types and draws threat intel from a 300k+ developer community, with contributors from OpenAI, Google, Microsoft, and Amazon. The workflow is code-first. A CLI command (npx promptfoo@latest redteam setup) wires up a scan, tests live in YAML, and runs slot into GitHub, GitLab, and Jenkins pipelines. Remediation guidance arrives in pull requests, so security work happens where developers already review code. Alongside Red Teaming the product spans real-time Guardrails, Model Security, an MCP Proxy for securing Model Context Protocol traffic, Code Scanning for VS Code and JetBrains, and LLM Evaluations. Industry packages cover financial services (FINRA-aligned), insurance, telecommunications, and real estate (fair housing compliance). The vendor says 156 of the Fortune 500 use it. In March 2026 Promptfoo agreed to be acquired by OpenAI, with the open-source project continuing under its founders. Against alternatives, Lakera leans toward runtime guardrails while Braintrust and LangSmith are evaluation-first with less security depth — Promptfoo's bet is that adversarial testing belongs in the developer workflow, not a separate dashboard.
Behind the Verdict
Promptfoo's real differentiator is placement, not vocabulary. Most AI security products sell you a dashboard; Promptfoo sells you a YAML test file and a CI job. The CLI entry point (npx promptfoo@latest redteam setup) generates attacks against your actual target, and findings land as remediation guidance inside pull requests — which means the security review happens in the same place your engineers already argue about diffs. For teams with an existing AppSec practice, that's a much shorter path to adoption than standing up a new console nobody logs into. The attack surface it covers is broader than pure prompt injection. It tests insecure tool use in agents, business rule violations, PII leaks, and toxic content generation, and it ships industry packages for financial services (FINRA-aligned), insurance, telecom, and real estate (fair housing compliance). The vendor claims 50+ vulnerability types and says 156 of the Fortune 500 use the product — plausible given the open-source base, whose contributors the site names as coming from OpenAI, Google, Microsoft, and Amazon. The breadth beyond testing — real-time Guardrails, Model Security, an MCP Proxy, IDE code scanning — matters if you want one vendor for pre-deploy testing and runtime defense. Where it grinds: this is a developer tool, not a switch. You need someone comfortable editing YAML and wiring GitHub, GitLab, or Jenkins jobs. Teams that want a runtime guardrail they can turn on without authoring tests will find the setup heavier than they expect, and Lakera is the more natural conversation there. The free tier's 10k probes per month is real but finite — a probe is a single request to your target, so continuous scanning across many applications will push you into a custom-quoted Enterprise plan. Evaluation-first shops comparing prompts and RAG pipelines may find LangSmith or Braintrust a better primary tool with Promptfoo layered on for security. The ownership question is the one to price in. In March 2026 Promptfoo agreed to be acquired by OpenAI; the open-source project continues under its founders. That is reassuring for continuity of the OSS tool and worth a hard look if you are a competitor to OpenAI or have procurement rules about single-vendor ownership of a security control.
Researching Promptfoo? Get your full AI stack in 60 seconds.
Free, no signup — tell us your goal and get tools matched to your budget & existing stack.
Real-world workflow fit
Concrete scenarios for the personas Promptfoo actually fits — and what changes day-one when you adopt it.
You run npx promptfoo@latest redteam setup against your customer-support agent, generate application-specific attacks for prompt injection and PII leakage, and commit the YAML config alongside the agent code. A GitHub Actions job runs the suite on every pull request.
Outcome: Findings appear as remediation guidance inside the PR, so the engineer who changed the prompt sees the vulnerability before merge rather than a security team filing a ticket a week later.
You need FINRA-aligned testing evidence for a customer-facing advisory assistant, plus coverage of business rule violations and data leaks. You build a scan config with the financial services package and run it in your GitLab pipeline on a schedule.
Outcome: You get a repeatable, documented testing trail tied to specific commits instead of a one-off pentest report, and failures block the pipeline the same way a failing unit test would.
Several product teams each ship their own LLM feature. You standardize on Promptfoo, share plugin collections and target settings across teams via Enterprise organization-specific configurations, and use the centralized dashboard and continuous monitoring to see risk across all of them.
Outcome: One view of which applications have unresolved findings and which fixes have landed, with SSO and role-based access so each team sees its own scope.
Use Cases
- Block a prompt change in CI if semantic similarity drops below threshold on 50 test cases.
- Red-team an agent with dozens of context-aware attack strategies before launching a public feature.
- Compare two models on your actual test suite and pick the winner by measurable metrics.
- Prevent prompt regressions by tying every merge to a passing Promptfoo run.
- Run FINRA-aligned security testing for a financial services AI assistant.
- Check fair housing compliance in a real estate AI agent before it talks to customers.
- Scan for PII leaks and business rule violations in a production RAG application.
- Test whether a web-browsing agent follows malicious instructions or leaks data.
Models Under the Hood
as of 2026-10-16
Limitations
- The free Community tier caps red teaming at 10,000 probes per month, and a probe is a single request to your target — continuous or multi-application scanning will exhaust it.
- Beyond that you buy Enterprise, which is custom-quoted; the pricing page says enterprise pricing is customized based on team size and needs, so you need a sales conversation to get a number.
- Advanced vulnerability detection, remediation recommendations, the centralized guardrail dashboard, customized red teaming plugins, organization-specific attack profiles, saved target configurations, searchable scan history, team sharing, SSO, custom roles, webhooks, and the central dashboard all sit behind Enterprise — Community gets basic vulnerability scanning only.
- Integration is developer-oriented (CLI, GitHub/GitLab/Jenkins pipelines, self-hosting) rather than plug-and-play.
- Since March 2026 Promptfoo has agreed to be acquired by OpenAI; the open-source project continues under its founders, but buyers weighing long-term vendor independence should factor that in.
as of 2026-10-22
Verification history
We have re-verified Promptfoo 21 times since . Each pass re-reads the vendor's own pages and re-checks every listed field against that evidence; passes where nothing had changed are marked as such.
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
- — re-verified summary, description, our verdict, our analysis, pricing model, pricing tiers, features, integrations, who it suits, who should skip it
Showing the 6 most recent of 21 verification passes.
Free to cite with attribution — this page re-verifies continuously.
12-month cost
Project the real annual outlay, including the implied monthly cost when only an annual tier is published.
Vendor list price only. Add-on usage, seat overages, and contract minimums are surfaced under Hidden costs & gotchas.
Plans compared
For each published Promptfoo tier: who it actually fits, and what it adds vs. the previous tier. Cross-reference the cost calculator above for projected annual outlay.
Community
$0/mo
Ideal for
Individual developers and small teams who can self-host and want full evaluation features plus basic vulnerability scanning without a contract
What this tier adds
Free entry point: all LLM evaluation features, every model provider and integration, custom integration with your own app, and red teaming up to 10k probes per month — but only basic vulnerability scanning
Enterprise
Custom
Ideal for
Larger teams and organizations that want to continuously monitor risk in development and production across multiple LLM applications
What this tier adds
Adds advanced vulnerability detection, remediation recommendations, the centralized guardrail dashboard, customized red teaming plugins, organization-specific attack profiles, saved target configurations, searchable scan history, team sharing, SSO, custom roles, webhooks,
On-Premise
Custom
Ideal for
Organizations that need full control of their infrastructure and complete data isolation, such as regulated enterprises with residency requirements
What this tier adds
Adds deployment on your own infrastructure, complete data isolation, a dedicated runner, and an assigned deployment engineer on top of every Enterprise feature
Where the pricing makes sense
The company stage and team size where Promptfoo's pricing actually pencils out — and where peers do it cheaper.
Community is free forever and unusually complete — all evaluation features, every model provider, and red teaming up to 10k probes per month, self-hosted on your own infrastructure. That undercuts paid-per-seat evaluation tools like Braintrust or LangSmith for a solo developer or small team. The jump is steep: everything beyond basic scanning — advanced detection, the guardrail dashboard, SSO, webhooks, teams — is Enterprise, custom-quoted by team size and needs. Budget for a sales cycle, not a
Setup time & first value
How long it actually takes to get something useful out of Promptfoo — broken out by persona, not the marketing-page minute.
A developer can get a first scan running in under an hour: one npx command scaffolds the red team config, and the YAML is editable straight away. Expect a day or two to get the suite wired into GitHub, GitLab, or Jenkins with PR-level reporting. Enterprise rollout across multiple applications, with SSO and shared organization configs, is a project measured in weeks — and on-premise adds an
Switching to or from Promptfoo
How to bring data in from common predecessors and how to get it back out — written for the switcher, not the buyer.
- →From manual prompt test scripts: port your test cases into Promptfoo YAML and run them through the same CLI that drives red teaming.
- →From a one-off LLM pentest: convert the pentest's findings into repeatable attack configs and run them on every pull request instead of annually.
- →From LangSmith or Braintrust evaluations: keep evaluation coverage but add red teaming, guardrails, MCP proxy, and IDE code scanning in the same tool.
- ↗To Lakera: if your priority shifts to runtime guardrails rather than pre-deploy testing, Lakera's model is built around blocking at inference time.
- ↗To LangSmith or Braintrust: if evaluation becomes the primary need and security testing secondary, those are evaluation-first platforms with less security depth.
Integrations
Resources & Guides
Tutorials & Learning
YouTube returned 6 videos for “Promptfoo”, and we withheld 6: 6 could not be judged, because “Promptfoo” is a single word that other videos use for other things. We are showing none, because we could not prove any of them are about Promptfoo.
Official links
Featured Head-to-Head Comparisons
Popular in AI Governance & Guardrails
Mindgard
Mindgard automates AI red teaming with agent-native reconnaissance and exploit validation for production AI systems
Poolside AI
Poolside builds open-weight agentic coding models — Laguna XS 2.1 (33B) and Laguna S 2.1 (118B) — for code that stays inside your security
Olas Network
Co-own autonomous AI agents that trade, hire each other and settle payments on-chain with OLAS staking.
Frequently Asked Questions
Best-of guides
Used Promptfoo? Help shape our editorial sentiment research.