v0.20.0Iris says what it checked. A verdict tells a pass from a check that never ran, names who recorded the evidence it judged, and cannot be replaced by the agent it judges→

Comparison · Evaluation

Iris vs Judgment Labs

MCP-Native Agent Eval vs SDK-Traced Hosted Agent Judge Platform.

TL;DR

Iris is an MCP server your agent discovers and uses on connect — one config block, no SDK, one SQLite file, every rule's precision and recall published at iris-eval.com/proof. Judgment Labs — from its own pages, read 2026-10-01: Judgment Labs calls Judgment the continuous-improvement stack for agents: the open-source judgeval SDK (Apache-2.0, Python and TypeScript) traces agents into a hosted platform where Agent Judges and Code Judges score traces and track behaviors, with a hosted MCP server for coding agents. The company, Judgment, Inc. of San Francisco, announced $32M in Seed and Series A funding led by Lightspeed on 2026-05-12; self-hosting is listed as coming soon. (source)

Iris grades what an agent did with its tools — the trace, the answer, the cost — not whether an MCP server honours its own contract; a server test harness answers that question, and Iris runs beside it. For the method, see the agent eval guide.

Feature comparison

Side by side.

13 features, the same 13 on every comparison. Every Judgment Labs cell links the page it was read from and the date; where the sentence it was read from was found verbatim on a plain download of that page (checked 2026-09-21), the link carries it as its title, and where it was not, the cell says so. The highlighted cells are Iris's own call on which side is stronger for a team running MCP agents — 4 to Iris, 2 to Judgment Labs — not a measurement.

FeatureIrisJudgment Labs
Integration methodOne block in the MCP config, no code — the agent discovers Iris and its tools on connectPython/TypeScript SDK (install judgeval): Tracer.init, Tracer.wrap on provider clients, Tracer.observe on custom codeJudgment Labs's page · read 2026-10-01
Self-hostingOne process, one SQLite file; Docker image with a health checkHosted platform only; self-hosting is listed as coming soonJudgment Labs's page · read 2026-10-01
Where it runsNothing in the agent's process — Iris is a separate server the agent callsIn-process SDK built on OpenTelemetry, initialized in every process that creates spans; exports to JudgmentJudgment Labs's page · read 2026-10-01
Evaluation26 built-in deterministic rules and 9 custom-rule types, in-process; 7 judge templates on a key you supply; every rule's precision and recall publishedAgent Judge follows a natural-language rubric; Code Judge runs your Python; outputs tracked as behaviors online/offlineJudgment Labs's page · read 2026-10-01
Cost trackingPer-trace USD cost and tokens; a cost spike judged against the agent's own historyProvider instrumentation captures token usage and cost; the traces table exposes a cost column per trace and spanJudgment Labs's page · read 2026-10-01
MCP supportProtocol-native — Iris is an MCP server with 12 tools; OTLP traces inHosted MCP server (mcp.judgmentlabs.ai) with OAuth for Cursor, Claude Code, Windsurf, Codex; tools query traces, create judges, run tests and evaluate tracesJudgment Labs's page · read 2026-10-01
LicenseMIT, the whole packagejudgeval SDK is Apache-2.0; the Judgment platform is hosted and proprietary (self-hosting listed as coming soon)Judgment Labs's page · read 2026-10-01
OwnershipIndependent and founder-ledJudgment, Inc. (Judgment Labs), San Francisco; VC-backed, $32M across Seed and Series A led by Lightspeed, announced 2026-05-12Judgment Labs's page · read 2026-10-01
DashboardA local dashboard on its own port: traces, moments, regressions, five viewsWeb UI: Traces table (previews, exceptions, judges, duration, cost), Conversation/Tree views, Judges and Behaviors pagesJudgment Labs's page · read 2026-10-01
Framework supportAny MCP client (2 verified, 8 claimed — see /clients); OTLP/HTTP from anything elseLangGraph, OpenAI Agents SDK, Claude Agent SDK, Google ADK, Vercel AI SDK; OpenAI, Anthropic, Gemini, Bedrock, LiteLLMJudgment Labs's page · read 2026-10-01
Prompt managementNot includedVersioned prompts on the platform with commit IDs and tags (e.g. production), fetched via the SDK; CLI deprecatedJudgment Labs's page · read 2026-10-01
Enterprise and complianceSelf-hosted. Nothing leaves your machine unless you turn one of these on: an OpenTelemetry endpoint (IRIS_OTEL_ENDPOINT), which exports traces to the collector you name; the LLM judge with your own key, which sends the text it judges to that provider, and whose citation check fetches the pages an output cites; or a webhook, which posts ids, the verdict and rule names, never the text, to the address you set. No compliance certification is claimed before it is heldNot stated in the vendor's documentation as of 2026-10-01Judgment Labs's page · read 2026-10-01
Cost to runFree — MIT, one process on your machine; the only spend is a judge call on a key you supply, when you opt inNot stated in the vendor's pages read on 2026-10-01Judgment Labs's page · read 2026-10-01

Decision guide

Which one fits your stack?

When to choose Iris

  • You are building with MCP-compatible agents and want the integration to be one config block
  • You want the evaluation to be deterministic and local — no model call, nothing leaving the machine
  • You want to read what each rule is worth before you trust it: every rule's precision and recall is published
  • You want self-hosting to be one process and one file
  • You want a fully permissive MIT license on the whole package

When to choose Judgment Labs

  • You need Agent Judges that apply a natural-language rubric to whole agent trajectories, alongside Code Judges written in Python. source
  • You need production behavior monitoring with detection rates per behavior and alerts when an agent regresses. source
  • You need auto-instrumentation for LangGraph, OpenAI Agents SDK, Claude Agent SDK, Google ADK or Vercel AI SDK with minimal code changes. source
  • You need OpenTelemetry-based tracing with dual export to Judgment and to your existing OTEL backend. source
  • You need offline tests that compare judge and agent versions across a fixed dataset built from production traces. source

FAQ

The questions buyers ask.

What is the difference between Iris and Judgment Labs?
Iris is an MCP-native agent eval server that needs no SDK. Add the config block, restart your client, and every session lists Iris's tools on connect. Iris never intercepts: it runs when your agent calls one of its tools, when a host hook or `iris-eval ingest` hands it a trace, or when you POST one to its HTTP API. Judgment Labs describes Judgment as the continuous-improvement stack for agents: its open-source judgeval SDK (Python and TypeScript, Apache-2.0) traces your agent into a hosted platform where Agent Judges and Code Judges score traces and track behaviors. source
Is Iris better than Judgment Labs for MCP agent evaluation?
Judgment offers things Iris does not: rubric-driven Agent Judges over full trajectories, behavior monitoring with regression alerts, offline tests over datasets built from production traces, and a hosted MCP server that lets coding agents query traces and judges. source
How much does Judgment Labs cost to run?
Iris: Free — MIT, one process on your machine; the only spend is a judge call on a key you supply, when you opt in. Judgment Labs's pages, read 2026-10-01, do not state it. source
Can I self-host Judgment Labs?
Iris: One process, one SQLite file; Docker image with a health check. Judgment Labs, from its own page read 2026-10-01: Hosted platform only; self-hosting is listed as coming soon. source
Does Judgment Labs work with MCP agents?
Iris: Protocol-native — Iris is an MCP server with 12 tools; OTLP traces in. Judgment Labs, from its own page read 2026-10-01: Hosted MCP server (mcp.judgmentlabs.ai) with OAuth for Cursor, Claude Code, Windsurf, Codex; tools query traces, create judges, run tests and evaluate traces. source

Sources

Every Judgment Labs statement on this page was read from one of these pages on the date shown. The file behind this page is website/src/lib/compare/judgment.json.

Last verified: 2026-10-01. This comparison is based on publicly available documentation and may not reflect recent changes to Judgment Labs. We aim to keep this page accurate and fair.

See something outdated or incorrect? Report an inaccuracy — we review and update within 48 hours.

Ready to see what your agents are doing?

Add Iris to your MCP config. First trace in 60 seconds. No SDK, no signup, no infrastructure.

Install IrisTry the playgroundEvery comparison