Promptfoo

Sweet spot: an engineering team that treats prompts and agent configs as code and wants the same CI-gated discipline they use for application code. Promptfoo fits exactly there — YAML configs live in the repo, assertions are precise, reports are diffable, and a GitHub Action can block regressions automatically. For teams shipping prompt changes weekly, it pays for itself the first time it catches a regression. Failure modes. It is not a hosted dashboard for non-developers — if the person iterating on prompts is not comfortable editing YAML in a repo, Braintrust or a hosted alternative will frustrate them less. LLM-as-judge evals are expensive at scale; keep test suites focused and sample aggressively. Red-teaming features are real but defence-in-depth — pair with human security review for high-stakes launches. What to pilot. Pick one production prompt you have recently regressed on. Write 20 Promptfoo test cases covering the regressions and the golden paths. Wire Promptfoo into CI. Next time someone edits the prompt, check whether the CI run catches expected failures before merge. If yes, expand the test suite; if no, the assertions need work before you can trust the harness.

Promptfoo is a developer-first evaluation framework for LLM applications. Its primary interface is a CLI and a YAML / TS config file: you declare a set of prompts, a set of test cases (with input variables and expected assertions), and a set of providers to run them against. Promptfoo runs everything in parallel and produces a diffable report you can compare across commits. The assertions library is comprehensive — equality, contains, regex, semantic-similarity, LLM-as-judge, classifier, factual consistency, latency budgets. You can define custom assertions as TypeScript / Python functions. Providers include every major LLM and custom HTTP endpoints. Beyond evaluations, Promptfoo also supports red-teaming: adversarial prompt generation, jailbreak detection, and a catalog of canonical attack patterns to test an agent against. This has made it a common pick in security-conscious AI teams. MIT-licensed, installable via npm / pip, and has a growing ecosystem. The maintainers offer a paid enterprise tier (promptfoo.dev) with team features, a cloud UI, and managed red-team runs, but the CLI and OSS features are sufficient for most teams.