# Iris — stop shipping agents on vibes

> Iris is an open-source MCP server that scores AI agent output quality, catches safety failures (PII leaks, prompt injection, hallucination markers), and enforces cost budgets. Any MCP-compatible agent or client discovers and uses it automatically — one command (`npx @iris-eval/mcp-server`), no SDK, no code changes. MIT-licensed core, self-hosted, signed releases.

Current release: v0.16.0 (2026-09-22) — Earn the comparison.

Iris registers 12 MCP tools (trace logging, heuristic eval, LLM-as-judge, semantic citation verification, rule lifecycle) and ships 25 built-in eval rules across completeness, relevance, safety, and cost. 9 of those rules read the agent's own tool calls and judge what it DID, not only what it wrote: a failed call the answer never acknowledges, a repeated one, a call its own tool's schema rejects, a file or URL cited that appears in nothing the agent read, an instruction obeyed out of a tool result, and a step budget. A ninth custom rule type, action_policy, states which tools an agent may call and with what arguments. All of it is deterministic and runs locally — no model call, nothing leaves the machine. It is verified on every CI run in Claude Code; for Claude Desktop, Cursor, Windsurf, Continue, VS Code, Cline, Zed, OpenAI Codex CLI, and Gemini CLI the installer writes the configuration shape each client documents and that writer is tested on the shape, with nobody on the Iris side having watched the client connect (claimed — every row with its source and date at https://iris-eval.com/clients); and it runs in any other MCP client the same way.

Evaluator accuracy is published at https://iris-eval.com/proof: precision, recall and F1 with 95% confidence intervals for 25 built-in rules, corpus 6a01ef7f62f8, generated 2026-09-22 from the source these numbers were measured on; reproduce with `npm run proof`.

Of 60 capability cells (10 evaluation questions by 6 subjects), 30 are answered by a shipped, measured thing, 22 are answered with a stated limit, 4 are open gaps and 4 do not apply — every answered cell names the rule, tool, resource, route, proof row or judge template behind it. The full map: https://iris-eval.com/capabilities.

## Getting Started

- [GitHub repository](https://github.com/iris-eval/mcp-server): source, README with per-client setup for every major MCP client, full tool schemas
- Claude Code plugin: `/plugin marketplace add iris-eval/mcp-server` then `/plugin install iris-eval@iris-eval` — MCP server + agent-eval skill in two commands
- [npm package](https://www.npmjs.com/package/@iris-eval/mcp-server): `@iris-eval/mcp-server` — install via `npx @iris-eval/mcp-server`
- [Playground](https://iris-eval.com/playground): try agent eval in the browser, no install
- [Proof](https://iris-eval.com/proof): how accurate the built-in evaluators are — method, per-rule results with confidence intervals, and the command to reproduce them
- [Capabilities](https://iris-eval.com/capabilities): what Iris can judge and what it cannot yet — ten questions by six subjects, each answered cell naming its evidence
- [Security](https://iris-eval.com/security): security posture, signed releases, disclosure policy, measured issue-close times
- [OTel recipes](https://github.com/iris-eval/mcp-server/blob/main/docs/otel-recipes.md): one recipe per framework — Pydantic AI, Google ADK, LangGraph via LangSmith, CrewAI via OpenInference, AutoGen, Microsoft Agent Framework, Semantic Kernel, Vercel AI SDK, Mastra — the vendor's own setup lines with Iris as the OTLP endpoint, what Iris reads from each vocabulary, and the fixture in the repository that proves it

## Learn — Agent Eval Concepts

- [What is Agent Eval](https://iris-eval.com/learn/agent-eval): the category definition
- [Eval Gap](https://iris-eval.com/learn/eval-gap): the distance between what you test and what your agents do
- [Eval Tax](https://iris-eval.com/learn/eval-tax): the cost of not evaluating
- [Eval Drift](https://iris-eval.com/learn/eval-drift): silent quality decay over time
- [Eval Coverage](https://iris-eval.com/learn/eval-coverage): how much of agent behavior your rules observe
- [Eval Loop](https://iris-eval.com/learn/eval-loop): the improvement cycle
- [Eval-Driven Development](https://iris-eval.com/learn/eval-driven-development): evals as the spec
- [Output Quality Score](https://iris-eval.com/learn/output-quality-score): how scoring works
- [Self-Calibrating Eval](https://iris-eval.com/learn/self-calibrating-eval): rules that adapt

## Compare

- [Iris vs Langfuse](https://iris-eval.com/compare/langfuse)
- [Iris vs LangSmith](https://iris-eval.com/compare/langsmith)
- [Iris vs Helicone](https://iris-eval.com/compare/helicone)
- [Iris vs Braintrust](https://iris-eval.com/compare/braintrust)
- [Iris vs Arize](https://iris-eval.com/compare/arize)
- [Iris vs DeepEval](https://iris-eval.com/compare/deepeval)
- [Iris vs Confident AI](https://iris-eval.com/compare/confident-ai)
- [Iris vs Patronus AI](https://iris-eval.com/compare/patronus-ai)

## Optional

- [Blog](https://iris-eval.com/blog): release notes, tutorials, agent eval essays
- [Pricing](https://iris-eval.com/pricing)
- [MCP Registry entry](https://registry.modelcontextprotocol.io): `io.github.iris-eval/mcp-server`
- [Docker image](https://github.com/iris-eval/mcp-server/pkgs/container/mcp-server): `ghcr.io/iris-eval/mcp-server`
- [llms-full.txt](https://iris-eval.com/llms-full.txt): expanded version of this file with tool reference and setup snippets