v0.20.0Iris says what it checked. A verdict tells a pass from a check that never ran, names who recorded the evidence it judged, and cannot be replaced by the agent it judges→

Comparison · Testing

Iris vs Promptfoo

MCP-Native Agent Eval vs Config-Driven CLI Eval and Red-Teaming.

TL;DR

Iris is an MCP server your agent discovers and uses on connect — one config block, no SDK, one SQLite file, every rule's precision and recall published at iris-eval.com/proof. Promptfoo — from its own pages, read 2026-10-01: Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM apps. Evals run on your machine and talk directly with the LLM, with results in a local web viewer. (source)

Iris grades what an agent did with its tools — the trace, the answer, the cost — not whether an MCP server honours its own contract; a server test harness answers that question, and Iris runs beside it. For the method, see the agent eval guide.

Feature comparison

Side by side.

13 features, the same 13 on every comparison. Every Promptfoo cell links the page it was read from and the date; where the sentence it was read from was found verbatim on a plain download of that page (checked 2026-09-22), the link carries it as its title, and where it was not, the cell says so. The highlighted cells are Iris's own call on which side is stronger for a team running MCP agents — 3 to Iris, 3 to Promptfoo — not a measurement.

FeatureIrisPromptfoo
Integration methodOne block in the MCP config, no code — the agent discovers Iris and its tools on connectOpen-source CLI and Node library driven by declarative YAML test cases; also runs in CI/CD.Promptfoo's page · read 2026-10-01
Self-hostingOne process, one SQLite file; Docker image with a health checkRuns locally; a basic Docker image hosts a server that stores evals (SQLite); not recommended for production.Promptfoo's page · read 2026-10-01
Where it runsNothing in the agent's process — Iris is a separate server the agent callsNo in-app instrumentation for output checks; evals run locally and call the LLM. Trajectory assertions read OTel spans the app sends. MCP Proxy is in-path.Promptfoo's page · read 2026-10-01
Evaluation26 built-in deterministic rules and 9 custom-rule types, in-process; 7 judge templates on a key you supply; every rule's precision and recall publishedDeterministic assertions including trajectory and tool-call assertions, plus model-graded (LLM rubric) assertions; custom JavaScript, Python and Ruby assertionsPromptfoo's page · read 2026-10-01
Cost trackingPer-trace USD cost and tokens; a cost spike judged against the agent's own historyA cost assertion checks provider-reported cost against a threshold; requires the provider to return cost information.Promptfoo's page · read 2026-10-01
MCP supportProtocol-native — Iris is an MCP server with 12 tools; OTLP traces inMCP provider tests MCP servers directly; ships an MCP server exposing its eval tools; Enterprise MCP Proxy product.Promptfoo's page · read 2026-10-01
LicenseMIT, the whole packageOpen-source core is MIT licensed; Promptfoo Enterprise (hosted SaaS) and Enterprise On-Prem are commercial.Promptfoo's page · read 2026-10-01
OwnershipIndependent and founder-ledAnnounced 2026-03-09 that it agreed to be acquired by OpenAI; its about page states Promptfoo is now part of OpenAI.Promptfoo's page · read 2026-10-01
DashboardA local dashboard on its own port: traces, moments, regressions, five viewsLocal web viewer opens eval results in the browser; results can be shared to promptfoo.app or a self-hosted server.Promptfoo's page · read 2026-10-01
Framework supportAny MCP client (2 verified, 8 claimed — see /clients); OTLP/HTTP from anything elseProviders for OpenAI, Anthropic, Azure, Google, HuggingFace, local models, HTTP, custom JS/Python scripts, MCP, n8n.Promptfoo's page · read 2026-10-01
Prompt managementNot includedPrompts inline or in Git-tracked files; external stores (Langfuse, Portkey, Helicone) referenced by version or label.Promptfoo's page · read 2026-10-01
Enterprise and complianceSelf-hosted. Nothing leaves your machine unless you turn one of these on: an OpenTelemetry endpoint (IRIS_OTEL_ENDPOINT), which exports traces to the collector you name; the LLM judge with your own key, which sends the text it judges to that provider, and whose citation check fetches the pages an output cites; or a webhook, which posts ids, the verdict and rule names, never the text, to the address you set. No compliance certification is claimed before it is heldEnterprise tier lists SSO, granular permission profiles, RBAC and teams; site shows SOC2 and ISO 27001 badges.Promptfoo's page · read 2026-10-01
Cost to runFree — MIT, one process on your machine; the only spend is a judge call on a key you supply, when you opt inCommunity edition free forever (open source; red teaming 10k probes a month); Enterprise by salesPromptfoo's page · read 2026-10-01

Decision guide

Which one fits your stack?

When to choose Iris

  • You are building with MCP-compatible agents and want the integration to be one config block
  • You want the evaluation to be deterministic and local — no model call, nothing leaving the machine
  • You want to read what each rule is worth before you trust it: every rule's precision and recall is published
  • You want self-hosting to be one process and one file
  • You want a fully permissive MIT license on the whole package

When to choose Promptfoo

  • You need red-teaming probes and vulnerability scanning of your LLM app alongside evals. source
  • You need declarative YAML test cases that run as a CLI step in CI/CD and compare models side-by-side. source
  • You need evals that run entirely on your machine and talk directly to the LLM, with no server. source
  • You need a proxy that governs which MCP servers your organization's applications and users may reach. source
  • You need an on-premises enterprise deployment with a dedicated runner behind your firewall. source

FAQ

The questions buyers ask.

What is the difference between Iris and Promptfoo?
Iris is an MCP-native agent eval server that needs no SDK. Add the config block, restart your client, and every session lists Iris's tools on connect. Iris never intercepts: it runs when your agent calls one of its tools, when a host hook or `iris-eval ingest` hands it a trace, or when you POST one to its HTTP API. Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM apps: tests are declared in config, run locally against the LLM, and viewed in a local web viewer. source
Is Iris better than Promptfoo for MCP agent evaluation?
Promptfoo offers red-teaming and vulnerability scanning, side-by-side model comparison across many providers, and an enterprise MCP Proxy that controls which MCP servers an organization can reach. source
How much does Promptfoo cost to run?
Iris: Free — MIT, one process on your machine; the only spend is a judge call on a key you supply, when you opt in. Promptfoo, from its own page read 2026-10-01: Community edition free forever (open source; red teaming 10k probes a month); Enterprise by sales. source
Can I self-host Promptfoo?
Iris: One process, one SQLite file; Docker image with a health check. Promptfoo, from its own page read 2026-10-01: Runs locally; a basic Docker image hosts a server that stores evals (SQLite); not recommended for production. source
Does Promptfoo work with MCP agents?
Iris: Protocol-native — Iris is an MCP server with 12 tools; OTLP traces in. Promptfoo, from its own page read 2026-10-01: MCP provider tests MCP servers directly; ships an MCP server exposing its eval tools; Enterprise MCP Proxy product. source

Sources

Every Promptfoo statement on this page was read from one of these pages on the date shown. The file behind this page is website/src/lib/compare/promptfoo.json.

Last verified: 2026-10-01. This comparison is based on publicly available documentation and may not reflect recent changes to Promptfoo. We aim to keep this page accurate and fair.

See something outdated or incorrect? Report an inaccuracy — we review and update within 48 hours.

Ready to see what your agents are doing?

Add Iris to your MCP config. First trace in 60 seconds. No SDK, no signup, no infrastructure.

Install IrisTry the playgroundEvery comparison