How often the evaluators themselves are wrong

How to read it

Each rule is run over a labelled corpus of agent outputs: precision is the share of outputs the rule flagged that were truly violations, recall is the share of true violations the rule flagged, and F1 is their harmonic mean. The intervals are 95% confidence intervals (wilson-95 for precision and recall, bootstrap-percentile-2000-mulberry32-seed-proof-f1-bootstrap-v1 for F1); a wide bar means a small sample, not a bad rule, and a rule whose interval reaches low is one whose verdict you should not gate a deploy on alone.

Two kinds of rule, two kinds of number. Where a rule’s documented definition is a formula (an output length, a coverage count, a keyword overlap, a cost, a token ratio), the corpus checks that the implementation matches its definition, so a precision and recall near 1.0 there mean “implemented as documented”, not “catches what goes wrong in the wild”. Where the definition names a judgement (PII, prompt injection, hallucination signals, stub answers, blocklisted phrases, staying on topic), the corpus measures detection against cases written to evade it, and those are the numbers to weigh before trusting a verdict.

Where the corpus came from

The disclosures that matter, stated before the numbers are read:

• Synthetic. The cases are constructed, not sampled from production traffic. They cover the shapes the rules are meant to catch; they do not tell you the base rate in your agent.
• LLM-authored. A language model wrote the cases, so they share one author’s idea of what a violation looks like.
• Same-model-labelled. The labels were produced by the same model family that authored the cases, which means the labels share the author’s blind spots. A rule that agrees with those labels agrees with that model, not yet with a human.
• Human blind label: pending. founder blind label of a 140-case stratified sample (twenty per judgment family); until then the labels are same-model dual annotation (see proof/README.md)

The verdict, measured

A gate does not key on a rule; it keys on passed, the verdict the composer makes from every rule that ran. The composite corpus scores that verdict: the 24 real transcripts, promoted as they are, plus cases composed by splicing a rule family’s case into a clean transcript so the failure classes present are true by construction. It scores the risk composer that has decided passed since 0.10.0 beside the arithmetic that decided it before, on the same rule results, so the change is readable rather than asserted.

Difference from legacy on the test split, risk composer with the per-output prior: 28.6 points [2.5, 49.8] (Newcombe 95%). An interval that straddles zero says the corpus cannot yet tell the two apart. Read per class, the plan’s prior blocks nearly every clean case; read per output it holds the legacy false-block rate — both readings are in the file, and which one ships is a decision the numbers inform. Cases: 119 (24 real transcripts, 95 composed; 91 dev / 28 test by a hash of the id). Composite ecc52d7452d9 · proof/COMPOSITE.md · proof/composite-results.json · npm run proof -- --composite.

The evasions a leak arrives in

The three critical rules match text. For every positive a rule caught with a span into the raw output, the text inside the span is transformed — a zero-width space, Cyrillic homoglyphs, fullwidth forms, a no-break space, a tab, a line break, swapped case — and the rule is run again. Recall per transform is the share still caught; a case the rule missed in the clear is not counted, and a transform that does not apply to a span (no letters to swap, no space to replace) is not counted for that case.

Per-transform intervals and the dropped case ids are in proof/RESULTS.md (results.json → transforms). The table measures the rules as shipped; a normalisation pass that changes it will change these numbers.

What no_pii finds, by entity

Every positive in the PII family names what it contains — by the case author, never by the detector. Per entity: present (cases containing it), caught (the rule failed the case for any reason), named (the rule’s evidence named this entity). The vocabulary includes things the rule’s definition does not cover, so a gap shows as a row with named 0 rather than as silence, and a case caught for another reason shows as the difference between caught and named.

Custom rule types — conformance

A custom rule is your own constraint, so its accuracy is whether the type does what its documented definition says under a declared config. One family per type, run through the same factory custom_rules and deployed rules go through; a disagreement here is a rule defect or a definition error, never an opinion.

Families: proof/corpus/custom/<type>.json. The json_schema family’s definition says the schema is not consulted in this version; its cases are labelled by that.

Evaluator of evaluators

Thirteen trust questions — does it work, when does it fail, what does it measure, is it calibrated, can it be gamed, can it produce false confidence, and seven more — asked of every evaluator Iris ships: the built-in rules, the custom rule types, the judge templates, the citation verifier and the verdict composer. Every cell is derived from the committed proof files by a generator, never typed; a cell reads measured only when a number for it exists and the evidence names the file.

Evaluators

33

Questions each

13

≥ 3 questions measured

27 / 33

• the built-in rules: 18 of 18 with three or more questions measured.
• the custom rule types: 8 of 8 with three or more questions measured.
• the LLM-judge templates: 0 of 5 with three or more questions measured — measurable, pending a judge key that you or the maintainer supplies.
• the citation verifier: 0 of 1 with three or more questions measured — measurable, pending a judge key that you or the maintainer supplies.
• the verdict composer: 1 of 1 with three or more questions measured.

The full matrix with the evidence per cell: docs/evaluators.md. Regenerated by npm run claims:generate and npm run llms:render at every release.

What is not in the table

The LLM-judge path (evaluate_with_llm_judge) and the citation verifier (verify_citations) are semantic, model-backed, and not deterministic; they are measured separately, on an adversarial set, under a cost cap. Status: pending. run npm run proof:judge with a judge API key, or dispatch the proof-judge workflow

Run it yourself

The corpus, the runner and the results file live in the repository. The numbers on this page are the committed results; regenerate them and diff:

git clone https://github.com/iris-eval/mcp-server.git
cd mcp-server
npm ci
npm run proof          # writes proof/results.json
git diff proof/results.json

No network and no API key are needed: the built-in rules are deterministic, so the run completes offline and identical inputs produce identical figures. The page is rendered from the same results file through the truthbase (.claims.json), and the hardcoded-claim scanner refuses any public sentence that says “measured” without linking here.

Related

• Security — the controls in place, and measured issue-close times.
• Roadmap — Track 1 is this page; what comes after it.
• docs/roadmap.md — the measurement commitments in full, including the ones not yet met.