See what your AI agents are

Core MCP Server

3 tools, initial 12-rule library, SQLite storage, web dashboard, production security

Eval Sensitivity + Security Hardening

Smart rule exclusion, configurable thresholds, SQL whitelist, CSP headers, accessibility

Dashboard Phase-1 + Pricing

OKLCH palette, dark/light theme, trace-ID copy, eval sparkline, pricing page, MCP-native validation harness

Rule Library Expansion

rule library grown to 13 (added no_stub_output), 10 PII patterns (IBAN, DOB, MRN, IP, API key, passport), 13 injection patterns, fabricated-citation heuristic, 55-case CI regression gate

LLM-as-Judge + Citation Verify + OTel + 9-tool MCP Surface

9 MCP tools — full rule + trace lifecycle + LLM-as-judge + SSRF-guarded citation verification (list_rules, deploy_rule, delete_rule, delete_trace, evaluate_with_llm_judge, verify_citations added); LLM-as-judge eval (Claude/GPT-4o, cost-capped, 5 prompt templates); semantic citation verification (4 citation kinds — numbered/author-year/URL/DOI — SSRF-guarded fetch + per-claim LLM verdict); OpenTelemetry export; tenant-id scaffolding; SBOM + cosign signing; Playwright E2E; Lighthouse CI; v2.C chrome polish

Security + Data Integrity

Dashboard bound to loopback with a DNS-rebinding guard; two ReDoS vectors closed (built-in PII patterns, and a deploy-time backtracking probe that catches what safe-regex2 misses); evaluations no longer dropped from time windows by a timestamp-format mismatch; the rule store no longer discards valid rules when one is unparseable; safety violations now counted even when the overall eval passes

The Acceptance-Test Release

Critical-rule veto — a detected PII leak, prompt injection or blocklist hit forces passed:false regardless of the weighted score and names itself in critical_failures; ReDoS sandbox worker with a hard 100ms per-match deadline plus a 3-breach-per-evaluation circuit breaker; strict tool arguments, so an unrecognised key fails the call instead of silently changing what gets evaluated; the safety-rule family measured against a labeled corpus and rebuilt (vendor-credential detection, structural injection detectors, context-grounded hallucination signals); POST /api/v1/traces HTTP ingest; --demo and --self-test; failure-first dashboard, now started only when explicitly enabled

That corpus was private and in-sample — the public per-rule numbers live on the proof page →

The Correctness Release

Every open item from the v0.5.0 acceptance pass closed and proven by a 52-row pre-release acceptance run: evaluate_output eval_type "all" runs every bundle in one pass with a per-category breakdown; deployed rules can be paused and resumed without deleting them; a browser can sign in to an --api-key dashboard; --version and --purge; retention sweeps stored evaluations; same-name deploy_rule refused unless replace: true; impossible get_traces ranges refused naming the values; the release workflow publishes to the Official MCP Registry and verifies npm, GHCR, the GitHub release and the registry from outside before it reports green.

The Proof Release

Every built-in rule got a public precision, recall and F1 with 95% confidence intervals, measured on a labelled corpus that ships in the repository and regenerates with one command — and a CI job that fails if the committed numbers stop matching the code. Acceptance ran 24 transcripts from an agent genuinely doing tasks against this codebase; nine defects surfaced and were fixed with regression tests before the cut. The judge and citation numbers stayed honestly marked pending, because measuring them needs a key we do not assume.

Every number on the proof page is regenerated by that command →

The Trajectory Release

The evaluator started reading the agent's tool calls, not only its answer: an answer written after a tool had silently failed, and a call repeated until it wasted the budget. Which rules VETO became the deployment's call rather than ours, matched by rule identity and reported on every response — because a precision lower bound of 77% cannot justify blocking everyone's work.

Iris Explains Its Verdicts

Every rule result now says what kind of claim it makes, what it looked at, where it found what it found as offsets into the raw text, and how wrong that kind of finding tends to be. Every response carries which layer decided, which questions were judged and which were not, and the versions and hashes that produced it. The server describes itself to an agent on connection, nine of nine tools carry an output schema, and every error returns a code with recovery steps instead of a sentence.

The Verdict

Iris stopped deciding by averaging. A composer reads each rule by the kind of claim it makes — a policy you configured gates, a high-precision detector vetoes, a critical check that was asked and could not answer makes the verdict UNKNOWN rather than clean, and the rest becomes one probability weighed against the loss ratio your deployment states. Measured before it shipped: right about shipping 57.7% of the time against the old 38.5%, at an identical false-block rate. Also: the critical text rules stopped being defeated by an invisible character, and the judge's own threshold decides instead of the model.

Both composers scored side by side on a held-out split →

The Act Layer

What the agent DID became evaluable, deterministically and without a model call: a call its own tool's schema rejects and the agent never retried, a file or URL cited that appears in nothing the agent read, an ask whose parts went unanswered, an instruction obeyed out of a tool result, and a step budget — plus action_policy, a custom rule type stating which tools an agent may call and with what arguments. Repetition is now caught as repeated sequences and as one target read through several tools, and a steady cadence is recognised as polling rather than a loop. One planned change was measured against a bar set beforehand, failed it, and was abandoned with both results published.

Including the change we measured and did not ship →

Proof — measure our own evaluators

Shipped: per-rule precision, recall and F1 with 95% intervals on a labelled corpus that regenerates with one command and is byte-checked in CI (v0.7.0); a labelled corpus for the ship verdict itself with a frozen held-out split, calibration, a threshold sweep, adversarial transforms and precision at four field prevalences (with v0.10.0); the blind-sample instrument for human agreement (v0.8.0). Still open: the human label on that sample; the judge's and the citation verifier's own numbers, which run on a key you supply; chance-corrected agreement.

The proof page — every number there is regenerated by the command it names →

Coverage — evaluate what actually fails

Shipped: the context-grounded hallucination rule (v0.5.0); the first trajectory rules — a silently failed tool call, a loop (v0.8.0); argument validity against the agent's own tool catalogue, grounding the output's file and URL citations in what the tools returned, multi-part ask coverage, injection compliance across a trajectory, a step ceiling and action_policy (v0.11.0). Still open: verification auditing (did the agent check its own work) and trace ingestion via OpenTelemetry GenAI semantic conventions.