What Iris can judge, and what it cannot yet

Deterministic detectors for leaked personal data and credentials, injection-shaped output and deployment-prohibited words, each fire naming the pattern and the span; precision and recall are published per rule at https://iris-eval.com/proof, and for personal data by entity.

needs: output

The safety detectors read the output; supplying the input changes nothing for them, so the single-output answer carries over. A keyed judge template for harm potential exists and is unmeasured.

needs: output

A directive found in a tool result is related to what the agent did next: no_injection_compliance matches directive wording as literal phrases over a capped, normalised slice — never a regular expression, because a tool result is attacker-controlled — and fails when a later call carries terms the directive introduced and the ask never used. Merely repeating those terms is reported on a passing result, because measured on its own family that weaker signal cannot tell reporting an injection from complying with one.

needs: outputtool_calls

Safe on nine of ten runs is readable for THIS question: compare_traces takes `question: safe_output` and counts only evaluations that judged it, an attempt passing when every rule answering it passed — a case's rate with a 95% Wilson interval, the flaky flag, and the run-level rate over cases with an exact interval, of the question rather than the composed verdict.

Per-rule failure counts over a time window are served to the dashboard, with no interval and no cohort key.

The safety detectors are measured on a labelled corpus with 95% intervals, and every fire carries its positive predictive value at the stated prior. Recall is measured again with the matched text disguised — a zero-width space, a look-alike letter, full-width letters, a no-break space, a tab, a line break, a change of case — and the cases each disguise drops are named.

Without an input only two context-free fabrication signals run, and the result says so; the grounded signals need the material the agent was given.

needs: output

A deterministic detector with context-grounded contradiction signals, measured with intervals; keyed judge templates for accuracy, correctness against a reference and faithfulness to sources, plus a citation verifier, cover what patterns cannot — on a key the user supplies.

needs: outputinput

Tool outputs are read as grounding material: a file, directory or URL the output cites that neither the ask nor anything the agent read mentions is evidence, with the span. Only locations are judged — a code identifier or a number belongs to the hallucination signals — and the rule declines rather than guesses when a read was truncated.

needs: outputinputtool_callstool_outputs

Consistency of claims across repeated runs of one input is not evaluated.

No cohort or baseline; only per-rule failure rates over time.

The fabrication detector is measured on its corpus; the judge templates are unmeasured until a user-supplied key runs the judge harness, and the page says pending until then.

Form is measured — length, sentence count, non-empty, stub or placeholder markers, and an answer that says nothing (a promise or a bare claim of completion in place of the work, a placeholder, an announcement with nothing after it, the ask handed back, one passage on a loop) — not whether the ask was answered.

needs: output

Coverage of an expected answer's terms is measured when one is supplied. Which of the parts an ask declares were answered is judged under the completion question, by ask_coverage; an ask written as flowing prose is not split into parts.

needs: outputexpected

The trajectory against the one the caller expected: tool_sequence compares the calls made with expected_trajectory.tool_calls in a mode (strict, unordered, subset, superset, ordered_subset — the default, a two-pointer subsequence check), matching by name and, when the expectation names arguments, by exact or subset input; a policy, so it gates because the caller stated the expectation; measured with intervals. Which parts of the ASK were acted on, without an expectation to compare against, is still the judge's question.

needs: tool_callsexpected_trajectory

Complete on most runs is readable for THIS question: compare_traces takes `question: complete` and counts only evaluations that judged it, an attempt passing when every rule answering it passed — a case's rate with a 95% Wilson interval, the flaky flag, and the run-level rate over cases with an exact interval, of the question rather than the composed verdict.

Per-rule failure counts over time only.

The completeness rules are measured as conformance to their stated formula, and each result says so in its uncertainty basis rather than presenting the number as a detection rate.

Both relevance rules skip without the input; on a single output the question is unanswerable and the verdict's coverage says so.

needs: outputinput

Lexical by default, semantic when the deployment sets a relevance judge. keyword_overlap (the share of the input's content terms the output carries) and topic_consistency (continuity of topic across sentences) are measurements that score; answers_the_ask is the question itself. Without a judge it reads the pair lexically and advises at the shipped thresholds, because comparing words fails correct paraphrases: it fires when both measurements fail and on a bare refusal or the ask handed back, and gates once the deployment sets either measurement's threshold. With IRIS_RELEVANCE_JUDGE_MODEL and a provider key, the relevance judge template decides and the rule gates: an off-topic answer fails on policy_gate, a paraphrase, a one-word answer and a one-term ask are judged rather than skipped, and a judge that cannot answer is named while the lexical reading advises. The lexical rule is measured on 45 reader-labelled cases; the judged rule and the relevance template are measured by the keyed proof:judge run and are pending until one is published.

needs: outputinput

The wrong tool for the ask, from the catalogue valid_tool_arguments already takes: tool_choice scores the ask's content terms against each tool's name, description and input property names (the relevance tokenizer) and fires when an uncalled tool fits the ask at least 0.5 and beats the union of the tools called by at least 0.34, naming both; judged over the whole trajectory so several tools that each serve part of the ask are not wrong; a heuristic detection, so it degrades the score and never vetoes; measured with intervals. Lexical: a tool whose description never mentions the ask's words is invisible to it.

needs: inputtool_callstools_catalogue

On-task on most runs is readable for THIS question: compare_traces takes `question: relevant` and counts only evaluations that judged it, an attempt passing when every rule answering it passed — a case's rate with a 95% Wilson interval, the flaky flag, and the run-level rate over cases with an exact interval, of the question rather than the composed verdict.

Per-rule failure counts over time only.

The two relevance measurements are measured as conformance to their lexical formula. answers_the_ask is measured on cases labelled by reading the ask and the answer, so its row shows where the lexical reading and a reader disagree; its judged form is unmeasured until a keyed run is published.

From the output alone, completion is a judgment: the task_completed judge template (0.15.0) reads the ask and the output — and the trajectory when the caller passes it — and scores whether the claims of completion are borne out. User-keyed; its measurement is pending until a key runs proof:judge, and the evaluators matrix says so.

needs: output

An ask that declares its parts — a bullet list, a numbered or lettered enumeration, a first/second/finally sequence — is checked part by part, and an unaddressed part is named with a span into the ask itself. A multi-part ask written as flowing prose is not split: a writer who numbers their questions is declaring separate things, and a full stop declares nothing.

needs: outputinput

A part of the ask the answer never mentions may still have been DONE: ask_coverage (v2, 0.15.0) reads the trajectory and counts a part covered when a tool call the trace shows carries its terms — the same lexical test as the answer, so the trajectory can only add coverage (the rule needs the output and the ask; it reads the trajectory when the trace carries one). Recall-only for the family; what the trajectory does not carry is still unjudged, and the judge template above is the other half.

needs: outputinput

Completes the task seven of ten runs is readable for THIS question: compare_traces takes `question: task_completed` and counts only evaluations that judged it, an attempt passing when every rule answering it passed — a case's rate with a 95% Wilson interval, the flaky flag, and the run-level rate over cases with an exact interval, of the question rather than the composed verdict (0.15.0).

No cohort or baseline for completion: nothing ships that says what an agent, a run or a period usually completes, so a drop in completion has no reference to be a drop against. Named as the one open cell of this question.

The ask-coverage reading of completion — were the parts of the ask addressed — has a published family (precision and recall per rule at https://iris-eval.com/proof); no end-to-end task-completion evaluator exists, so nothing measures completion beyond that reading.

Acting well is a property of the trajectory, not of a single output.

As for a single output.

Three deterministic trajectory rules, each measured with intervals: a failed tool call the output never acknowledges, a repeated call with the same input, and a call whose arguments the tool’s own JSON Schema rejects and the agent never retried. Argument validity needs the tools catalogue; tool CHOICE — whether a valid call was the right one — is judged in the on-task row (Q4×S3, tool_choice).

needs: outputtool_callstools_catalogue

Acts well on most runs is readable for THIS question: compare_traces takes `question: tool_use_correct` and counts only evaluations that judged it, an attempt passing when every rule answering it passed — a case's rate with a 95% Wilson interval, the flaky flag, and the run-level rate over cases with an exact interval, of the question rather than the composed verdict.

Per-rule failure counts for the two trajectory rules over time only.

Both trajectory rules are measured with intervals on their corpus families, and each fire carries its positive predictive value.

Trace cost and token usage are captured and judged by the cost bundle, and the evaluator's own spend — the judge's tokens and cost — is stored on the evaluation.

needs: cost

A cost ceiling and a token ratio are judged from the trace totals; the ratio measures verbosity, not efficiency, and says so.

needs: costtokens

A step budget is judged from the trajectory itself: max_steps fails a task that takes more tool calls than the deployment's ceiling, advising at the shipped default and gating once a deployment sets its own; step_budget fails a task that takes more than the budget the CALLER stated for it (expected_trajectory.step_budget, or the number of expected calls) times a tolerance, and gates because the caller stated it; no_tool_loop counts the repeated work inside either. Per-call tokens and cost are carried on a call and not yet read; the trace TOTAL is judged elsewhere in this row.

needs: tool_callsexpected_trajectory

Within budget across repeats of one input is readable as the case's pass rate under the cost rules: compare_traces takes `question: within_budget` and counts only evaluations that judged it — a case's rate with a 95% Wilson interval and a flaky flag, and the run-level rate over cases with an exact interval. The spread of the cost itself across repeats of one input is not reported: cost_anomaly reads an agent's cost history, not a case's.

Aggregate cost over a time window is served to the dashboard, and a trace is judged against the agent's own cost distribution: the cost_anomaly rule and the cost-spike moment read the modified z over the agent's last 200 costed traces (median and MAD), never a fixed dollar line.

needs: cost

The cost rules are measured as conformance to their formula, and the evaluator's own spend is stored per evaluation.

Better or worse than before needs at least two things to compare.

As for a single output.

A trajectory regression shows as its rule failing more often between two runs, reported per rule, worst first, once traces are logged with tool calls and a run id. The trajectories themselves are not compared step by step.

Evaluations of the same case across runs are grouped by case key and reported as a per-case pass rate with a 95% interval; a case answered both ways is reported as flaky. The run-level rate counts each case once, with an exact interval, so ten repeats of one question are not counted as ten questions.

Two runs are compared on pass rate with a 95% interval on the difference; when the runs share case keys they are paired and the discordant cases are tested exactly. When the evidence cannot exclude "no change" it says so and reports the smallest change that many cases could have detected. Runs measuring different things are refused unless forced, naming what differs. Scores bucketed over time still feed the drift view, and a CUSUM per agent and rule watches the stream itself: a regression-alarm moment marks the evaluation at which a rule's fail rate crossed a line derived for an in-control run length of about 500, reporting and never gating. The discordant cases are listed with both evaluations and the rules that flipped, and a run can be pinned as the baseline every later run is compared against (PATCH /api/v1/runs/:id; an omitted before means the baseline).

The verdict a gate keys on is measured on a composite corpus, and a candidate composer is scored beside the shipped one with an interval on the difference and a held-out split, so a change to the composer is measured before it ships. On a deployment’s own traces the same question is answerable directly: re-scoring a run under the current rules writes the new verdicts into a new run marked as a re-evaluation, and comparing the two isolates the rules change — the executions are identical, so any difference is the evaluator.

Every fired detection carries typed evidence — offsets into the raw output, never a paraphrase — and a message naming the pattern; the stored evaluation reads back the same way.

needs: output

The grounded signals name the contradiction they found; measurements carry their value, unit and the threshold they were held to and where it came from.

needs: outputinput

A silent tool failure names the failed call by index, why it failed and what the output claimed instead; a loop names every repeated call and the threshold.

needs: outputtool_calls

Where it fails across repeats: a case answered both ways is reported as flaky, least reliable first. compare_traces groups every evaluation of one case across runs and reports the case's pass rate with a 95% Wilson interval and a flaky flag, and the run-level rate over cases with an exact interval — of the composed verdict, or of one evaluation question with `question`. The rate says how often a case fails, not why: the reasons are on each evaluation, and nothing gathers them across the repeats.

Top failures over a window are served to the dashboard; the held-out check is 24 staged transcripts from one agent on one repository, reported without an interval; there is no sample of unstaged production traffic.

Every false positive and false negative per rule is named by case id with the proof; the safety detectors that report a span are measured again under each evasion transform applied inside that span, with the cases each transform drops; the PII rule reports recall by entity, including the entities its definition does not cover; the judge templates stay unmeasured until a user-supplied key runs the judge harness.

Every rule result carries what kind of claim it is, what it saw, where in the raw text it found it, and how wrong it tends to be; the verdict names which layer decided — a configured policy, a high-precision detector, a check that could not answer, or the combined risk against the deployment's stated loss ratio — and one definition of passed covers the deterministic rules, the judge and the citation verifier alike.

needs: output

The verdict names which layer decided it and which rules did the deciding, coverage says which evaluation questions were judged and which input was missing for the rest, and a critical check that was asked and could not answer makes the verdict unknown rather than clean — a state a gate can fail closed on, and a setting the deployment chooses. Every rule deploy, delete, toggle and update and every trace deletion is recorded in an audit log the agent can read, so a rule changed or evidence removed around a verdict can be found afterwards.

needs: outputinput

When tool calls are absent the trajectory rules skip and coverage names the question as unjudged; both rules are non-critical by default, so their skip never vetoes.

needs: outputtool_calls

Trust over repeats is a case's pass rate with a 95% Wilson interval and a flaky flag, and a run-level rate over cases with an exact interval, so ten repeats of one question are not ten questions; the interval is the trust statement. No composer votes over repeats — the rate is reported, not turned into one verdict — and the judge has no repeat harness.

Population-level trust: two windows are compared with both denominators and a 95% interval on the difference in pass rate, shown on the dashboard's change banner; two runs are compared with a Newcombe interval on the difference (McNemar on the discordant pairs when the cases pair). An interval that straddles zero is reported as not distinguishable, with the sample that would distinguish it. The deployment's own labels on a rule's fires say how often those fires were right on its traffic, with a Wilson interval; at twenty labels that number is in force on every verdict (the risk estimate reads it in place of the published one), the prior is estimated from the rule's fire rate, and the rule whose next label narrows the most uncertainty is named.

Precision, recall and intervals per rule are published and reproducible with one command, served to agents as a resource and on every rule result as its uncertainty; beside them, what a fire is worth at field prevalence, a credible interval that does not collapse at zero errors, the verdict’s own accuracy on a composite corpus, conformance of every custom rule type to its definition, and a matrix that asks thirteen trust questions of every evaluator and answers each cell from the proof files; the corpus is model-labelled until the blind human label lands, and every number says so.