Proof
How often the evaluators themselves are wrong
Every eval tool tells you your agent’s score. This page tells you how much to trust the scorer: each built-in rule’s precision, recall and F1, with the uncertainty shown rather than hidden, on a corpus you can download, from a command you can run.
Generated October 4, 2026 for v0.20.0 · corpus 7c27824a2b98 · schema v2
- Measured for detection
- 13 rules
- Checked against own formula
- 13 rules
- Labelled cases
- 1,323
- Corpus
- 7c27824a2b98
Two kinds of number, never added together. 13 rules are measured against labels a model gave by reading the failure itself, without running the rule (a synthetic, model-labelled corpus; a human blind label is pending): their precision and recall say how well the rule detects it (1 of 13 score F1 1.00). 13 rules are checked against their own documented definition, applied independently by script or by counting: a score there shows the code implements its formula, not that the formula catches what a reader would call the failure (12 of 13 score F1 1.00). Each row says which.
Bar = 95% confidence interval · tick = point estimate · scale 0 to 1. n splits into labelled violations (+) and labelled clean outputs (−); tp, fp, fn, tn are the confusion counts the three figures are computed from. PPV at 1% · 5% is what a fire is worth when one output in a hundred, or five, is a violation: the corpus is about half positive, and a deployment rarely is, so the precision column overstates a fire at field prevalence.
How long one evaluation takes
- Median (p50)
- 0.837 ms
- 95th percentile
- 5.117 ms
EvalEngine.evaluateAll — the call evaluate_output makes — over every case in the proof corpus, 25 warm-up runs discarded, storage excluded. Over 1323 evaluations on 12th Gen Intel(R) Core(TM) i9-12900HK (win32/x64, node v24.11.0). Your machine is not this machine: the number is here so there is one to check rather than a word like “fast”, and it is re-measured on every proof run.
completeness
| Rule | n | Precision | Recall | F1 | PPV at 1% · 5% |
|---|---|---|---|---|---|
| min_output_length checked against its own formula | 24 = 13 + / 11 − tp 13 · fp 0 · fn 0 · tn 11 | 1.00 0.77–1.00 | 1.00 0.77–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| non_empty_output measured against labels from reading the failure | 28 = 14 + / 14 − tp 12 · fp 0 · fn 2 · tn 14 | 1.00 0.76–1.00 | 0.86 0.60–0.96 | 0.92 0.78–1.00 | 1.00 · 1.00 |
| sentence_count checked against its own formula | 24 = 14 + / 10 − tp 14 · fp 0 · fn 0 · tn 10 | 1.00 0.78–1.00 | 1.00 0.78–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| expected_coverage checked against its own formula | 24 = 11 + / 13 − tp 11 · fp 0 · fn 0 · tn 13 | 1.00 0.74–1.00 | 1.00 0.74–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| valid_tool_arguments checked against its own formula | 33 = 15 + / 18 − tp 15 · fp 0 · fn 0 · tn 18 | 1.00 0.80–1.00 | 1.00 0.80–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| ask_coverage measured against labels from reading the failure | 30 = 13 + / 17 − tp 11 · fp 4 · fn 2 · tn 13 | 0.73 0.48–0.89 | 0.85 0.58–0.96 | 0.79 0.58–0.93 | 0.04 · 0.16 |
| tool_sequence checked against its own formula | 51 = 28 + / 23 − tp 28 · fp 0 · fn 0 · tn 23 | 1.00 0.88–1.00 | 1.00 0.88–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| says_something measured against labels from reading the failure | 242 = 116 + / 126 − tp 23 · fp 0 · fn 93 · tn 126 | 1.00 0.86–1.00 | 0.20 0.14–0.28 | 0.33 0.22–0.43 | 1.00 · 1.00 |
relevance
| Rule | n | Precision | Recall | F1 | PPV at 1% · 5% |
|---|---|---|---|---|---|
| keyword_overlap checked against its own formula | 24 = 11 + / 13 − tp 11 · fp 0 · fn 0 · tn 13 | 1.00 0.74–1.00 | 1.00 0.74–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| topic_consistency checked against its own formula | 24 = 12 + / 12 − tp 11 · fp 0 · fn 1 · tn 12 | 1.00 0.74–1.00 | 0.92 0.65–0.99 | 0.96 0.84–1.00 | 1.00 · 1.00 |
| tool_choice measured against labels from reading the failure | 46 = 20 + / 26 − tp 18 · fp 0 · fn 2 · tn 26 | 1.00 0.82–1.00 | 0.90 0.70–0.97 | 0.95 0.86–1.00 | 1.00 · 1.00 |
| answers_the_ask measured against labels from reading the failure | 45 = 20 + / 25 − tp 20 · fp 3 · fn 0 · tn 22 | 0.87 0.68–0.95 | 1.00 0.84–1.00 | 0.93 0.84–1.00 | 0.08 · 0.30 |
safety
| Rule | n | Precision | Recall | F1 | PPV at 1% · 5% |
|---|---|---|---|---|---|
| no_pii measured against labels from reading the failure | 101 = 49 + / 52 − tp 44 · fp 5 · fn 5 · tn 47 | 0.90 0.78–0.96 | 0.90 0.78–0.96 | 0.90 0.83–0.96 | 0.09 · 0.33 |
| no_blocklist_words measured against labels from reading the failure | 26 = 15 + / 11 − tp 15 · fp 1 · fn 0 · tn 10 | 0.94 0.72–0.99 | 1.00 0.80–1.00 | 0.97 0.89–1.00 | 0.10 · 0.37 |
| no_injection_patterns measured against labels from reading the failure | 97 = 45 + / 52 − tp 44 · fp 1 · fn 1 · tn 51 | 0.98 0.88–1.00 | 0.98 0.88–1.00 | 0.98 0.94–1.00 | 0.34 · 0.73 |
| no_stub_output measured against labels from reading the failure | 93 = 44 + / 49 − tp 32 · fp 5 · fn 12 · tn 44 | 0.86 0.72–0.94 | 0.73 0.58–0.84 | 0.79 0.68–0.88 | 0.07 · 0.27 |
| no_hallucination_markers measured against labels from reading the failure | 90 = 46 + / 44 − tp 34 · fp 0 · fn 12 · tn 44 | 1.00 0.90–1.00 | 0.74 0.60–0.84 | 0.85 0.76–0.92 | 1.00 · 1.00 |
| no_silent_tool_failure measured against labels from reading the failure | 74 = 38 + / 36 − tp 29 · fp 0 · fn 9 · tn 36 | 1.00 0.88–1.00 | 0.76 0.61–0.87 | 0.87 0.77–0.94 | 1.00 · 1.00 |
| grounded_in_reads measured against labels from reading the failure | 30 = 14 + / 16 − tp 14 · fp 0 · fn 0 · tn 16 | 1.00 0.78–1.00 | 1.00 0.78–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| no_injection_compliance measured against labels from reading the failure | 31 = 14 + / 17 − tp 9 · fp 0 · fn 5 · tn 17 | 1.00 0.70–1.00 | 0.64 0.39–0.84 | 0.78 0.54–0.95 | 1.00 · 1.00 |
cost
| Rule | n | Precision | Recall | F1 | PPV at 1% · 5% |
|---|---|---|---|---|---|
| cost_under_threshold checked against its own formula | 24 = 10 + / 14 − tp 10 · fp 0 · fn 0 · tn 14 | 1.00 0.72–1.00 | 1.00 0.72–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| verbosity_ratio checked against its own formula | 24 = 12 + / 12 − tp 12 · fp 0 · fn 0 · tn 12 | 1.00 0.76–1.00 | 1.00 0.76–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| no_tool_loop checked against its own formula | 37 = 18 + / 19 − tp 18 · fp 0 · fn 0 · tn 19 | 1.00 0.82–1.00 | 1.00 0.82–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| max_steps checked against its own formula | 28 = 12 + / 16 − tp 12 · fp 0 · fn 0 · tn 16 | 1.00 0.76–1.00 | 1.00 0.76–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| cost_anomaly checked against its own formula | 27 = 13 + / 14 − tp 13 · fp 0 · fn 0 · tn 14 | 1.00 0.77–1.00 | 1.00 0.77–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
| step_budget checked against its own formula | 46 = 22 + / 24 − tp 22 · fp 0 · fn 0 · tn 24 | 1.00 0.85–1.00 | 1.00 0.85–1.00 | 1.00 1.00–1.00 | 1.00 · 1.00 |
How to read it
Each rule is run over a labelled corpus of agent outputs: precision is the share of outputs the rule flagged that were truly violations, recall is the share of true violations the rule flagged, and F1 is their harmonic mean. The intervals are 95% confidence intervals (wilson-95 for precision and recall, bootstrap-percentile-2000-mulberry32-seed-proof-f1-bootstrap-v1 for F1); a wide bar means a small sample, not a bad rule, and a rule whose interval reaches low is one whose verdict you should not gate a deploy on alone.
Two kinds of rule, two kinds of number. Where a rule’s documented definition is a formula (an output length, a coverage count, a keyword overlap, a cost, a token ratio), the corpus checks that the implementation matches its definition, so a precision and recall near 1.0 there mean “implemented as documented”, not “catches what goes wrong in the wild”. Where the definition names a judgement (PII, prompt injection, hallucination signals, stub answers, blocklisted phrases, staying on topic), the corpus measures detection against cases written to evade it, and those are the numbers to weigh before trusting a verdict.
Where the corpus came from
The disclosures that matter, stated before the numbers are read:
- Synthetic. The cases are constructed, not sampled from production traffic. They cover the shapes the rules are meant to catch; they do not tell you the base rate in your agent.
- LLM-authored. A language model wrote the cases, so they share one author’s idea of what a violation looks like.
- Same-model-labelled. The labels were produced by the same model family that authored the cases, which means the labels share the author’s blind spots. A rule that agrees with those labels agrees with that model, not yet with a human.
- Human blind label: pending. founder blind label of a 140-case stratified sample (twenty per judgment family); until then the labels are same-model dual annotation (see proof/README.md)
The verdict, measured
A gate does not key on a rule; it keys on passed, the verdict the composer makes from every rule that ran. The composite corpus scores that verdict: the 24 real transcripts, promoted as they are, plus cases composed by splicing a rule family’s case into a clean transcript so the failure classes present are true by construction. It scores the risk composer that has decided passed since 0.10.0 beside the arithmetic that decided it before, on the same rule results, so the change is readable rather than asserted.
| Split | Composer | Accuracy vs should-ship (95% CI) | False blocks on clean | Missed blocks |
|---|---|---|---|---|
| test | legacy | 35.3% [21.5, 52.1] n=34 | 10.0% n=10 | 87.5% n=24 |
| test | risk, per-output prior | 64.7% [47.9, 78.5] n=34 | 10.0% n=10 | 45.8% n=24 |
| real transcripts (held out, staged) | legacy | 45.8% [27.9, 64.9] n=24 | 0.0% n=6 | 72.2% n=18 |
| real transcripts (held out, staged) | risk, per-output prior | 70.8% [50.8, 85.1] n=24 | 0.0% n=6 | 38.9% n=18 |
Difference from legacy on the test split, risk composer with the per-output prior: 29.4 points [5.7, 48.9] (Newcombe 95%). An interval that straddles zero says the corpus cannot yet tell the two apart. Read per class, the plan’s prior blocks nearly every clean case; read per output it holds the legacy false-block rate — both readings are in the file, and which one ships is a decision the numbers inform. Cases: 145 (24 real transcripts — staged runs held out of every per-rule rate, not production traffic, as the next section says — and 121 composed; 111 dev / 34 test by a hash of the id). Composite e1d72ff465c1 · proof/COMPOSITE.md · proof/composite-results.json · npm run proof -- --composite.
Why τ stays at 0.50 when the sweep peaks at 0.15. The threshold sweep on the dev split (utility = −(false blocks + c × missed blocks) at c = 1 on the dev split; the argmax is a check on the loss model, never adopted) finds its utility-optimal cut at 0.15; the shipped cut is 0.50, which is 1 / (1 + eval.falsePassCost) at the shipped cost of 1. The arithmetic behind the gap: the per-output prior π = 0.50 is spread over the 16 failure classes an output can carry, so one class starts at πc = 1 − (1 − π)1/16 ≈ 0.042; a single fire from no_stub_output (sensitivity 72.7%, specificity 89.8% on the labelled corpus) then has a positive predictive value of about 24.0% — below τ, so one fire passes, and a corpus that is 69.0% must-not-ship rewards a lower cut.
Established: the sweep’s optimum is a fact about this corpus, not about the field; adopting it would fit the threshold to the cases it is scored on, and the prior is a stated assumption about a deployment we have never seen. The honest way to move the number is to let the deployment’s own prior in — eval.prior is a config key today, with its source printed on every result, and the local labels that estimate it are the next arc. From the bins: In 6 of the 7 populated reliability bins on the test split the observed bad rate sits above the mean predicted P(bad) (n-weighted gap +28.8 points): the risk estimate is under-confident on bad outputs. That is what the bins say, and it is the direction a local prior corrects.
The same idea under another name. A conformal prediction set for a two-valued verdict is either one label or both. Iris has the both-labels case in one of its forms: confidence: "marginal" with a credible interval on pbad that straddles τ is exactly the set {pass, fail}. The analogy stops there. A verdict is also marginal where the reliability table above has not confirmed the estimate: too few labelled verdicts or detector patterns in its tenth of pbad, or an observed bad rate on the dev split that disagrees with the prediction or reaches τ. That is a statement about how far the estimate has been checked, not a prediction set, and it carries no coverage guarantee.
Held out: 24 real agent runs, staged
Every other number on this page is measured on a corpus written alongside the rule it measures. These are agent runs against this repository with an answer key written at capture time, and no labelled case behind any per-rule number above is copied from, varied from or modelled on one of them, so the verdict that scores them was not composed from rates estimated on them. Read what that does and does not buy before quoting a number from this table:
- Staged, not sampled. Each run was scripted with an intended failure: 18 of 24 are bad by design. That base rate is chosen, not observed, so these are not the error rates of any real deployment.
- Held out, not blind. Several rules were written or revised after an acceptance pass on these runs, so the rules have seen them even though the accuracy numbers have not.
- Small and narrow. Twenty-four traces from one agent on one repository, reported without an interval. There is no sample of unstaged production traffic yet.
| Question | Result |
|---|---|
| Failure classes present that some rule caught | 14 of 23 |
| Ship/no-ship verdicts agreeing with the key | 17 of 24 |
| Transcripts whose four bundle verdicts all agree | 13 of 24 |
| Bundle verdicts agreeing | 82 of 96 |
The three are not the same question. Classes caught is the one to read: it needs no relabelling as rules are added. A bundle verdict is a weighted mean, so one failing non-critical rule in a bundle of six does not move it — the weakest of the three, and kept for continuity. 11 of the 24 transcripts have at least one bundle verdict that disagrees with the key; those gaps are measured and read back by the test suite, so a gap that closes cannot stay recorded as open. Transcripts cc79a661de5f · proof/TRANSCRIPTS.md · proof/transcript-results.json · npm run proof -- --transcripts.
The evasions a leak arrives in
The three critical rules match text. For every positive a rule caught with a span into the raw output, the text inside the span is transformed — a zero-width space, Cyrillic homoglyphs, fullwidth forms, a no-break space, a tab, a line break, swapped case — and the rule is run again. Recall per transform is the share still caught; a case the rule missed in the clear is not counted, and a transform that does not apply to a span (no letters to swap, no space to replace) is not counted for that case.
| Rule | zero width | homoglyph | fullwidth | nbsp | tab | linebreak | case |
|---|---|---|---|---|---|---|---|
| no_pii | 100.0% n=44 | 100.0% n=36 | 100.0% n=44 | 92.9% n=14 | 84.1% n=44 | 84.1% n=44 | 77.8% n=36 |
| no_injection_patterns | 100.0% n=44 | 100.0% n=44 | 100.0% n=44 | 86.5% n=37 | 90.9% n=44 | 88.6% n=44 | 97.7% n=44 |
| no_blocklist_words | 100.0% n=15 | 100.0% n=15 | 100.0% n=15 | 100.0% n=14 | 93.3% n=15 | 93.3% n=15 | 100.0% n=15 |
| no_injection_compliance | 100.0% n=9 | 100.0% n=9 | 100.0% n=9 | 100.0% n=9 | 100.0% n=9 | 100.0% n=9 | 100.0% n=9 |
Per-transform intervals and the dropped case ids are in proof/RESULTS.md (results.json → transforms). The table measures the rules as shipped; a normalisation pass that changes it will change these numbers.
What no_pii finds, by entity
Every positive in the PII family names what it contains — by the case author, never by the detector. Per entity: present (cases containing it), caught (the rule failed the case for any reason), named (the rule’s evidence named this entity). The vocabulary includes things the rule’s definition does not cover, so a gap shows as a row with named 0 rather than as silence, and a case caught for another reason shows as the difference between caught and named.
| Entity | Present | Caught | Named | Recall (95% CI) |
|---|---|---|---|---|
| ssn | 4 | 4 | 4 | 100.0% [51.0, 100.0] |
| credit_card | 2 | 2 | 2 | 100.0% [34.2, 100.0] |
| iban | 1 | 1 | 0 | 0.0% [0.0, 79.3] |
| phone | 7 | 7 | 7 | 100.0% [64.6, 100.0] |
| 12 | 12 | 12 | 100.0% [75.8, 100.0] | |
| dob | 2 | 2 | 2 | 100.0% [34.2, 100.0] |
| private_key | 2 | 2 | 2 | 100.0% [34.2, 100.0] |
| seed_phrase | 1 | 1 | 1 | 100.0% [20.6, 100.0] |
| api_key | 17 | 15 | 10 | 58.8% [36.0, 78.4] |
| password | 3 | 3 | 0 | 0.0% [0.0, 56.1] |
| address | 6 | 3 | 0 | 0.0% [0.0, 39.0] |
| url_token | 1 | 1 | 0 | 0.0% [0.0, 79.3] |
Custom rule types — conformance
A custom rule is your own constraint, so its accuracy is whether the type does what its documented definition says under a declared config. One family per type, run through the same factory custom_rules and deployed rules go through; a disagreement here is a rule defect or a definition error, never an opinion.
| Type | Config | n | Skipped | Precision (95% CI) | Recall (95% CI) |
|---|---|---|---|---|---|
| action_policy | {"allow":[{"args":{"/path":"/workspace/**"},"tool":"read_file"},{"args":{"/path":"/workspace/**"},"tool":"list_dir"},{"args":{"/url":"https://docs.vendor.test/**"},"tool":"web_fetch"}],"deny":[{"args":{"/path":"/workspace/.env*"},"tool":"read_file"},{"tool":"bash"},{"tool":"http_post"}]} | 30 | 0 | 100.0% [80.6, 100.0] | 100.0% [80.6, 100.0] |
| contains_keywords | {"keywords":["refund","policy","days"],"threshold":1} | 24 | 0 | 100.0% [78.5, 100.0] | 100.0% [78.5, 100.0] |
| cost_threshold | {"max_cost":0.05} | 24 | 2 | 100.0% [75.8, 100.0] | 100.0% [75.8, 100.0] |
| excludes_keywords | {"keywords":["guarantee","risk-free"]} | 24 | 0 | 100.0% [74.1, 100.0] | 100.0% [74.1, 100.0] |
| json_schema | {"schema":{"additionalProperties":false,"properties":{"count":{"type":"integer"},"id":{"type":"string"},"nested":{"properties":{"ok":{"type":"boolean"}},"required":["ok"],"type":"object"},"tags":{"items":{"type":"string"},"type":"array"}},"required":["id","count"],"type":"object"}} | 27 | 0 | 100.0% [79.6, 100.0] | 100.0% [79.6, 100.0] |
| max_length | {"max_length":120} | 24 | 0 | 100.0% [78.5, 100.0] | 100.0% [78.5, 100.0] |
| min_length | {"min_length":80} | 24 | 0 | 100.0% [77.2, 100.0] | 100.0% [77.2, 100.0] |
| regex_match | {"pattern":"^Ticket #[0-9]{6}\\b"} | 24 | 0 | 100.0% [78.5, 100.0] | 100.0% [78.5, 100.0] |
| regex_no_match | {"pattern":"\\bTODO\\b"} | 24 | 0 | 100.0% [75.8, 100.0] | 100.0% [75.8, 100.0] |
Families: proof/corpus/custom/<type>.json. The json_schema family’s definition says the schema is not consulted in this version; its cases are labelled by that.
Evaluator of evaluators
Thirteen trust questions — does it work, when does it fail, what does it measure, is it calibrated, can it be gamed, can it produce false confidence, and seven more — asked of every evaluator Iris ships: the built-in rules, the custom rule types, the judge templates, the citation verifier and the verdict composer. Every cell is derived from the committed proof files by a generator, never typed; a cell reads measured only when a number for it exists and the evidence names the file.
- Evaluators
- 44
- Questions each
- 13
- ≥ 3 questions measured
- 36 / 44
- the built-in rules: 26 of 26 with three or more questions measured.
- the custom rule types: 9 of 9 with three or more questions measured.
- the LLM-judge templates: 0 of 7 with three or more questions measured — measurable, pending a judge key that you or the maintainer supplies.
- the citation verifier: 0 of 1 with three or more questions measured — measurable, pending a judge key that you or the maintainer supplies.
- the verdict composer: 1 of 1 with three or more questions measured.
The full matrix with the evidence per cell: docs/evaluators.md. Regenerated by npm run claims:generate and npm run llms:render at every release.
Related: the capability map says what Iris can judge; this page says how well.
What is not in the table
The LLM-judge path (evaluate_with_llm_judge) and the citation verifier (verify_citations) are semantic, model-backed, and not deterministic; they are measured separately, on an adversarial set, under a cost cap. Status: pending. run npm run proof:judge with a judge API key, or dispatch the proof-judge workflow
Run it yourself
The corpus, the runner and the results file live in the repository. The numbers on this page are the committed results; regenerate them and diff:
git clone https://github.com/iris-eval/mcp-server.git cd mcp-server npm ci npm run proof # writes proof/results.json git diff proof/results.json
No network and no API key are needed: the built-in rules are deterministic, so the run completes offline and identical inputs produce identical figures. The page is rendered from the same results file through the truthbase (.claims.json), and the hardcoded-claim scanner refuses any public sentence that says “measured” without linking here.
Related
- Security — the controls in place, and measured issue-close times.
- Roadmap — Track 1 is this page; what comes after it.
- Capabilities — every question Iris can be asked about every subject, with what it lacks said in its own words.