How do you calculate the eval tax?

The eval tax compounds across three dimensions: direct costs (manual review hours, incident response), indirect costs (delayed shipping, lost developer confidence), and risk costs (undetected PII exposure, hallucinated answers reaching users). Most teams underestimate it because the costs are distributed across the organization.

How do you eliminate the eval tax?

By scoring every agent output inline, with deterministic rules that run in under a millisecond, so nothing stands between you and full coverage. Iris exposes scoring as an MCP tool the agent calls; traffic that must be scored regardless of what the model chooses goes through POST /api/v1/traces from your own code. When eval runs on 100% of outputs, the tax drops to near zero. The key is making eval effortless — if it requires pipelines or manual review, teams skip it and the tax compounds.

The Eval Tax

Q: What is the eval tax?

The eval tax is the compounding cost of every unscored agent output — measured in trust erosion, engineering hours spent on manual review, and liability exposure. Teams without agent eval pay this tax on every execution, whether they realize it or not.

The hidden, compounding cost every agent team pays when they don't evaluate output quality.

Definition#

Definition

The eval tax is the compounding cost of every unscored agent output — measured in trust erosion, engineering hours spent on manual review, and liability exposure. Every agent execution without evaluation adds to the balance. The longer you wait to start evaluating, the higher the accumulated debt.

How It Compounds#

The eval tax isn't a one-time cost — it compounds. Each unscored output that reaches a user without evaluation creates downstream effects: support tickets from hallucinated answers, engineering time spent debugging agent behavior manually, compliance risk from undetected PII in outputs. These costs grow exponentially as agent usage scales.

The insidious part: most teams don't realize they're paying it. The costs are distributed across support, engineering, and legal — never attributed back to the missing eval layer. Teams describe symptoms ("we spend too much time reviewing agent outputs") without connecting them to the root cause.

Three Dimensions of the Eval Tax#

Direct Costs

Manual review hours, incident response, output correction, customer support from bad agent answers.

Indirect Costs

Delayed shipping (fear of agent failures), lost developer confidence, slower iteration cycles.

Risk Costs

Undetected PII exposure, hallucinated answers in production, compliance violations, reputational damage.

How Iris Helps#

Iris removes the setup cost that keeps teams paying the eval tax: scoring is an MCP tool the agent calls inline, at the protocol layer. No SDK, no pipeline to build, no manual review. Add one line to your MCP config and the agent can score any output for quality, safety, and cost — and for traffic that must be scored whether or not the model chooses to, your code can send it to POST /api/v1/traces instead.

npx @iris-eval/mcp-server

# Ask the agent to evaluate an output, and Iris scores it for:
# - Completeness (response length, structure)
# - Relevance (topic consistency)
# - Safety (PII detection, prompt injection)
# - Cost (token usage, USD per trace)

Read the deep dive: The AI Eval Tax →

TERM

Eval Drift

Quality degradation over time — one of the mechanisms that drives up the eval tax.

TERM

Eval Coverage

The percentage of outputs being evaluated. 0% coverage = maximum eval tax.

TERM

The Eval Gap

The distance between demo and production — where the eval tax accumulates fastest.

TERM

Agent Eval