v0.20.0Iris says what it checked. A verdict tells a pass from a check that never ran, names who recorded the evidence it judged, and cannot be replaced by the agent it judges→

Comparison · Observability

Iris vs Helicone

MCP-Native Agent Eval vs API Gateway Observability.

TL;DR

Iris is an MCP server your agent discovers and uses on connect — one config block, no SDK, one SQLite file, every rule's precision and recall published at iris-eval.com/proof. Helicone — from its own pages, read 2026-10-01: Helicone is an open-source, proxy-based AI gateway and observability platform with cost analytics by custom property, response caching, and routing across 100+ models; acquired by Mintlify in March 2026, it now runs in maintenance mode. If you need per-property cost analytics, response caching, or a unified gateway across many LLM providers, Helicone covers that ground. (source)

Iris grades what an agent did with its tools — the trace, the answer, the cost — not whether an MCP server honours its own contract; a server test harness answers that question, and Iris runs beside it. For the method, see the agent eval guide.

Feature comparison

Side by side.

13 features, the same 13 on every comparison. Every Helicone cell links the page it was read from and the date; where the sentence it was read from was found verbatim on a plain download of that page (checked 2026-09-22), the link carries it as its title, and where it was not, the cell says so. The highlighted cells are Iris's own call on which side is stronger for a team running MCP agents — 3 to Iris, 3 to Helicone — not a measurement.

FeatureIrisHelicone
Integration methodOne block in the MCP config, no code — the agent discovers Iris and its tools on connectProxy-based (change base URL + add header)Helicone's page · read 2026-10-01
Self-hostingOne process, one SQLite file; Docker image with a health checkAll-in-one Docker image (web, API, PostgreSQL, ClickHouse, MinIO); Helm chart for KubernetesHelicone's page · read 2026-10-01
Where it runsNothing in the agent's process — Iris is a separate server the agent callsRust AI gateway; vendor-reported P95 latency under 5 msHelicone's page · read 2026-10-01
Evaluation26 built-in deterministic rules and 9 custom-rule types, in-process; 7 judge templates on a key you supply; every rule's precision and recall publishedNo built-in evaluators; accepts scores from external eval frameworks via API or the dashboardHelicone's page · read 2026-10-01
Cost trackingPer-trace USD cost and tokens; a cost spike judged against the agent's own historyCost per user, conversation, feature or environment via custom propertiesHelicone's page · read 2026-10-01
MCP supportProtocol-native — Iris is an MCP server with 12 tools; OTLP traces inMCP server for data access onlyHelicone's page · read 2026-10-01
LicenseMIT, the whole packageApache 2.0 for the platform; the separate Rust AI Gateway repository is GPL-3.0Helicone's page · read 2026-10-01
OwnershipIndependent and founder-ledAcquired by Mintlify (Mar 3, 2026); product in maintenance mode; last changelog entry Nov 26, 2025Helicone's page · read 2026-10-01
DashboardA local dashboard on its own port: traces, moments, regressions, five viewsDashboard with Requests, Sessions, Users, Segments and HQL query viewsHelicone's page · read 2026-10-01
Framework supportAny MCP client (2 verified, 8 claimed — see /clients); OTLP/HTTP from anything elseGateway integrations for LangChain, LangGraph, LlamaIndex, Vercel AI SDK, Zapier, CodexHelicone's page · read 2026-10-01
Prompt managementNot includedVersioned prompts with rollback and per-environment deployment via the AI GatewayHelicone's page · read 2026-10-01
Enterprise and complianceSelf-hosted. Nothing leaves your machine unless you turn one of these on: an OpenTelemetry endpoint (IRIS_OTEL_ENDPOINT), which exports traces to the collector you name; the LLM judge with your own key, which sends the text it judges to that provider, and whose citation check fetches the pages an output cites; or a webhook, which posts ids, the verdict and rule names, never the text, to the address you set. No compliance certification is claimed before it is heldSOC 2 Type II + HIPAA (Team and Enterprise), GDPR via EU region, SAML SSO, on-prem, Helm chartHelicone's page · read 2026-10-01
Cost to runFree — MIT, one process on your machine; the only spend is a judge call on a key you supply, when you opt inHobby free (10,000 requests a month, 1 GB storage, 1 seat); Pro $79/month; Team $799/month; usage-based pricing applies; Enterprise customHelicone's page · read 2026-10-01

Decision guide

Which one fits your stack?

When to choose Iris

  • You are building with MCP-compatible agents and want the integration to be one config block
  • You want the evaluation to be deterministic and local — no model call, nothing leaving the machine
  • You want to read what each rule is worth before you trust it: every rule's precision and recall is published
  • You want self-hosting to be one process and one file
  • You want a fully permissive MIT license on the whole package

When to choose Helicone

  • You need cost analytics — per-user, per-conversation, per-feature spend breakdowns via custom properties source
  • You want response caching to avoid repeat charges on identical requests source
  • You need a full AI gateway with retries, fallbacks, and rate limiting source
  • You need enterprise compliance today (SOC 2, GDPR) source
  • You want a unified gateway across 100+ models and 20+ LLM providers source

FAQ

The questions buyers ask.

What is the difference between Iris and Helicone?
Iris is an MCP-native agent eval server that needs no SDK. Add the config block, restart your client, and every session lists Iris's tools on connect. Iris never intercepts: it runs when your agent calls one of its tools, when a host hook or `iris-eval ingest` hands it a trace, or when you POST one to its HTTP API. Helicone is an open-source, proxy-based AI gateway and observability platform with cost analytics by custom property, response caching, and routing across 100+ models; it was acquired by Mintlify in March 2026 and is in maintenance mode. Helicone offers cost breakdowns and gateway features, and does not run evaluations itself. source
Is Iris better than Helicone for MCP agent evaluation?
Helicone uses a proxy-based approach (a Rust gateway with vendor-reported P95 latency under 5 ms) and offers response caching, rate limiting, fallbacks, and cost analytics by custom property; it does not run evaluations itself. source
How much does Helicone cost to run?
Iris: Free — MIT, one process on your machine; the only spend is a judge call on a key you supply, when you opt in. Helicone, from its own page read 2026-10-01: Hobby free (10,000 requests a month, 1 GB storage, 1 seat); Pro $79/month; Team $799/month; usage-based pricing applies; Enterprise custom. source
Can I self-host Helicone?
Iris: One process, one SQLite file; Docker image with a health check. Helicone, from its own page read 2026-10-01: All-in-one Docker image (web, API, PostgreSQL, ClickHouse, MinIO); Helm chart for Kubernetes. source
Does Helicone work with MCP agents?
Iris: Protocol-native — Iris is an MCP server with 12 tools; OTLP traces in. Helicone, from its own page read 2026-10-01: MCP server for data access only. source

Sources

Every Helicone statement on this page was read from one of these pages on the date shown. The file behind this page is website/src/lib/compare/helicone.json.

Last verified: 2026-10-01. This comparison is based on publicly available documentation and may not reflect recent changes to Helicone. We aim to keep this page accurate and fair.

See something outdated or incorrect? Report an inaccuracy — we review and update within 48 hours.

Ready to see what your agents are doing?

Add Iris to your MCP config. First trace in 60 seconds. No SDK, no signup, no infrastructure.

Install IrisTry the playgroundEvery comparison