What is the difference between Iris and DeepEval?

Iris is an MCP-native eval server that scores every agent output inline in production using deterministic heuristic rules. DeepEval is a Python testing framework that runs offline eval suites using LLM-as-Judge metrics via pytest. Iris requires zero code changes and runs in real-time. DeepEval requires Python test files and runs as a batch testing step.

Should I use Iris or DeepEval for agent evaluation?

Use Iris if you want inline production evaluation with zero setup — add one line to your MCP config and every agent output gets scored automatically. Use DeepEval if you need LLM-as-Judge evaluation with nuanced semantic scoring, are building in Python, and want to run eval suites in CI/CD pipelines.

Comparison

Iris vs DeepEval

Inline production eval vs offline test suites. Two different philosophies for evaluating AI agent output.

TL;DR

Iris is an MCP server that scores every agent output inline in production — zero code changes, deterministic rules, sub-millisecond overhead. DeepEval is a Python testing framework that runs LLM-as-Judge evaluation suites via pytest — powerful semantic metrics, designed for CI/CD pipelines. If you want every production output scored automatically, Iris. If you want deep semantic evaluation in your test pipeline, DeepEval.

For background on heuristic vs semantic evaluation, see our evaluation methodology guide.

Feature Comparison

Side by side.

Feature	Iris	DeepEval
Eval approach	Deterministic heuristic rules (<1ms)	LLM-as-Judge metrics (semantic, slower)
Integration method	MCP config (zero code)	Python pytest decorators
When eval runs	Inline, every output in production	Offline, batch test suites in CI/CD
Language	TypeScript (any MCP agent)	Python only
Self-hosting	Single binary, one SQLite file	pip install, local execution
Built-in metrics	12 rules (completeness, safety, cost, relevance)	14+ metrics (faithfulness, hallucination, bias, toxicity)
Custom metrics	Zod schema custom rules	Python custom metrics class
Cost tracking	Per-trace USD cost, aggregate visibility	Not included
Dashboard	Real-time dark-mode UI	Confident AI cloud dashboard (separate product)
MCP support	Protocol-native (IS an MCP server)	Not MCP-aware
License	MIT	Apache 2.0
Maturity	Early stage (v0.1.8)	Established (4K+ GitHub stars)

Decision Guide

Which one fits your workflow?

When to choose Iris

You want eval running on every output in production, not just in test suites
You're building with MCP-compatible agents and want zero-code integration
You need cost tracking and aggregate spend visibility across agents
You want a single self-hosted binary with no Python dependency
You want deterministic, fast, predictable scoring without LLM API costs

When to choose DeepEval

You need semantic evaluation with LLM-as-Judge (faithfulness, hallucination, bias)
You're building in Python and want pytest-native eval workflows
You want to run eval suites in CI/CD pipelines before deployment
You need a mature ecosystem with extensive documentation and community
You want the Confident AI cloud platform for team collaboration

Last verified: March 2026. This comparison is based on publicly available documentation and may not reflect recent changes to DeepEval. We aim to keep this page accurate and fair.

See something outdated or incorrect? Report an inaccuracy — we review and update within 48 hours.

Ready to see what your agents are doing?

Add Iris to your MCP config. First trace in 60 seconds. No SDK, no signup, no infrastructure.

Try Iris Join Cloud Waitlist