What Iris can judge, and what it cannot yet

Deterministic detectors for leaked personal data and credentials, injection-shaped output and deployment-prohibited words, each fire naming the pattern and the span; precision and recall are published per rule at https://iris-eval.com/proof, not yet per entity.

needs: output

The safety detectors read the output; supplying the input changes nothing for them, so the single-output answer carries over. A keyed judge template for harm potential exists and is unmeasured.

needs: output

Tool outputs are never scanned for injected directives, and no rule relates a directive in a tool result to a later action.

needs: outputtool_calls

No grouping over repeated runs of one input exists, so nothing can say safe on nine of ten runs.

Per-rule failure counts over a time window are served to the dashboard, with no interval and no cohort key.

The three safety detectors are measured on a labelled corpus with 95% intervals, and every fire carries its positive predictive value at the stated prior; robustness to obfuscated text is not yet measured.

Without an input only two context-free fabrication signals run, and the result says so; the grounded signals need the material the agent was given.

needs: output

A deterministic detector with context-grounded contradiction signals, measured with intervals; keyed judge templates for accuracy, correctness against a reference and faithfulness to sources, plus a citation verifier, cover what patterns cannot — on a key the user supplies.

needs: outputinput

Tool outputs are read as grounding material: a file, directory or URL the output cites that neither the ask nor anything the agent read mentions is evidence, with the span. Only locations are judged — a code identifier or a number belongs to the hallucination signals — and the rule declines rather than guesses when a read was truncated.

needs: outputinputtool_callstool_outputs

Consistency of claims across repeated runs of one input is not evaluated.

No cohort or baseline; only per-rule failure rates over time.

The fabrication detector is measured on its corpus; the judge templates are unmeasured until a user-supplied key runs the judge harness, and the page says pending until then.

Form is measured — length, sentence count, non-empty, and stub or placeholder markers — not whether the ask was answered.

needs: output

Coverage of an expected answer's terms is measured when one is supplied; nothing yet reads the structure of the ask to see which parts were answered.

needs: outputexpected

No completeness rule reads the trajectory to see which parts of the ask were acted on.

needs: outputtool_calls

No grouping over repeated runs.

Per-rule failure counts over time only.

The completeness rules are measured as conformance to their stated formula, and each result says so in its uncertainty basis rather than presenting the number as a detection rate.

Both relevance rules skip without the input; on a single output the question is unanswerable and the verdict's coverage says so.

needs: outputinput

Lexical: the share of the input's content terms the output carries, and continuity of topic across sentences, with a stated tokenizer and thresholds; a keyed helpfulness judge template exists and is unmeasured.

needs: outputinput

Nothing relates the trajectory to the ask, so a wrong tool for the question goes unjudged on every path.

needs: outputtool_calls

No grouping over repeated runs.

Per-rule failure counts over time only.

The relevance rules are measured as conformance to their lexical formula; the proof does not yet include cases where the formula and a reader would disagree.

Task completion cannot be judged from the output alone, and no rule attempts it.

needs: output

An ask that declares its parts — a bullet list, a numbered or lettered enumeration, a first/second/finally sequence — is checked part by part, and an unaddressed part is named with a span into the ask itself. A multi-part ask written as flowing prose is not split: a writer who numbers their questions is declaring separate things, and a full stop declares nothing.

needs: outputinput

No rule judges whether the trajectory completed the task; the real-transcript acceptance record names this as open.

needs: outputtool_calls

No grouping over repeated runs; completes the task seven of ten runs is not expressible.

No cohort or baseline.

No completion evaluator exists to measure.

Acting well is a property of the trajectory, not of a single output.

As for a single output.

Three deterministic trajectory rules, each measured with intervals: a failed tool call the output never acknowledges, a repeated call with the same input, and a call whose arguments the tool’s own JSON Schema rejects and the agent never retried. Argument validity needs the tools catalogue; tool CHOICE — whether a valid call was the right one — still has no deterministic evaluator.

needs: outputtool_callstools_catalogue

No grouping over repeated runs.

Per-rule failure counts for the two trajectory rules over time only.

Both trajectory rules are measured with intervals on their corpus families, and each fire carries its positive predictive value.

Trace cost and token usage are captured and judged by the cost bundle, and the evaluator's own spend — the judge's tokens and cost — is stored on the evaluation.

needs: cost

A cost ceiling and a token ratio are judged from the trace totals; the ratio measures verbosity, not efficiency, and says so.

needs: costtokens

Cost is judged on the trace total; per-call latency is recorded, per-call tokens and cost are not yet.

needs: cost

No grouping over repeated runs, and no rule reads an agent's own cost distribution.

Aggregate cost over a time window is served to the dashboard; the cost-spike classifier uses a fixed threshold rather than the agent's own distribution.

The cost rules are measured as conformance to their formula, and the evaluator's own spend is stored per evaluation.

Better or worse than before needs at least two things to compare.

As for a single output.

No comparison of trajectories across runs.

Nothing evaluates several runs of one input; the only canonicalisation of an input serves the loop detector.

Scores bucketed over time are served to the dashboard's drift view; no interval and no cohort key, so a change is shown, not tested.

The verdict a gate keys on is measured on a composite corpus, and a candidate composer is scored beside the shipped arithmetic with an interval on the difference and a held-out split, so a change to the composer is measured before it ships; nothing yet compares one release’s evaluator with the previous release’s on the same corpus across versions.

Every fired detection carries typed evidence — offsets into the raw output, never a paraphrase — and a message naming the pattern; the stored evaluation reads back the same way.

needs: output

The grounded signals name the contradiction they found; measurements carry their value, unit and the threshold they were held to and where it came from.

needs: outputinput

A silent tool failure names the failed call by index, why it failed and what the output claimed instead; a loop names every repeated call and the threshold.

needs: outputtool_calls

No grouping over repeated runs.

Top failures over a window are served to the dashboard; the out-of-sample check is a small set of real transcripts from one agent, reported without an interval.

Every false positive and false negative per rule is named by case id with the proof; the three critical rules are measured under seven evasion transforms inside their evidence span, with the cases each transform drops; the PII rule reports recall by entity, including the entities its definition does not cover; the judge templates stay unmeasured until a user-supplied key runs the judge harness.

Every rule result carries what kind of claim it is, what it saw, where in the raw text it found it, and how wrong it tends to be; the verdict names which layer decided — a configured policy, a high-precision detector, a check that could not answer, or the combined risk against the deployment's stated loss ratio — and one definition of passed covers the deterministic rules, the judge and the citation verifier alike.

needs: output

The verdict names which layer decided it and which rules did the deciding, coverage says which evaluation questions were judged and which input was missing for the rest, and a critical check that was asked and could not answer makes the verdict unknown rather than clean — a state a gate can fail closed on, and a setting the deployment chooses.

needs: outputinput

When tool calls are absent the trajectory rules skip and coverage names the question as unjudged; both rules are non-critical by default, so their skip never vetoes.

needs: outputtool_calls

A verdict over repeats — vote, mean, variance — has no composer, and the judge has no repeat harness.

No population-level trust statement; intervals on pass rates appear nowhere on the dashboard.

Precision, recall and intervals per rule are published and reproducible with one command, served to agents as a resource and on every rule result as its uncertainty; beside them, what a fire is worth at field prevalence, a credible interval that does not collapse at zero errors, the verdict’s own accuracy on a composite corpus, conformance of every custom rule type to its definition, and a matrix that asks thirteen trust questions of every evaluator and answers each cell from the proof files; the corpus is model-labelled until the blind human label lands, and every number says so.