Heuristic vs Semantic Eval: When <1ms Matters More Than LLM-as-Judge

There is a default assumption in the agent eval space right now: if you want to evaluate agent output, you need an LLM to judge it. Feed the output to GPT-4o with a rubric, get a score back, done. LLM-as-Judge is the pattern everyone reaches for first.

I want to push back on that. Not because LLM-as-Judge is bad -- it is genuinely powerful for certain problems. But because most teams are using it for evaluations that do not require an LLM at all. They are spending seconds and dollars on checks that a regex can handle in microseconds for free.

Two Approaches to Agent Evaluation

LLM-as-Judge sends your agent's output to another LLM with a scoring prompt. The judge model reads the output, compares it against criteria you define, and returns a score. This is semantic evaluation -- the judge understands meaning, nuance, and context.

Strengths: handles subjective quality, can assess factual accuracy against source documents, evaluates tone and style, reasons about complex multi-step outputs.

Weaknesses: adds 1-5 seconds of latency per evaluation, costs $0.01-0.05 per call depending on the model and output length, introduces non-determinism (run the same eval twice and you might get different scores), and requires managing yet another LLM integration.

Heuristic rules are pattern-based checks: regex matches, string comparisons, length calculations, threshold checks. They run against the output directly with no model inference.

Strengths: sub-millisecond execution, deterministic (same input always produces the same result), zero marginal cost, no external dependencies.

Weaknesses: cannot understand meaning, cannot assess subjective quality, limited to patterns you can express as rules.

The question is not which approach is better. The question is which problems actually need semantic understanding and which are better served by a fast, deterministic check.

When Heuristic Rules Win

Here are the evaluations I see teams running through LLM-as-Judge that do not need an LLM:

PII detection. Social Security numbers follow the pattern \d{3}-\d{2}-\d{4}. Credit card numbers are four groups of four digits. Phone numbers, email addresses -- these are structural patterns. A regex catches them in microseconds with zero ambiguity. You do not need a 70-billion-parameter model to determine whether a string matches \b\d{3}-\d{2}-\d{4}\b.

Prompt injection detection. The most common injection patterns are string-matchable: "ignore all previous instructions," "you are now a," "bypass your safety filters." These are not subtle. A set of regex patterns catches them deterministically:

const INJECTION_PATTERNS = [
  /ignore (?:all )?(?:previous|above|prior) (?:instructions|prompts)/i,
  /you are now (?:a |in )/i,
  /bypass (?:your |the )?(?:safety|content|ethical) (?:filters|guidelines|restrictions)/i,
];

Output length and completeness. Did the agent return an empty response? Is the output below a minimum character count? Does it contain at least one complete sentence? These are arithmetic checks.

Cost threshold enforcement. Did this trace cost more than $0.10? Is the completion-to-prompt token ratio above 5:1 (suggesting the agent is generating far more than it should relative to the input)? These are numeric comparisons.

Blocklist enforcement. You have a list of phrases that should never appear in agent output. Checking whether a string contains a substring is not a job for an LLM.

Every one of these evaluations is better served by a heuristic rule: faster, cheaper, deterministic, and easier to debug when something fails.

When Semantic Evaluation Wins

There are evaluations that genuinely need an LLM to judge:

Factual accuracy. Given a set of source documents, did the agent's response accurately represent the facts? This requires reading comprehension and reasoning about whether the output is consistent with the sources. Pattern matching cannot do this.

Tone and style assessment. Is this customer support response empathetic? Is this technical explanation at the right level for the audience? Tone is subjective and context-dependent. You need a model that understands language pragmatics.

Complex reasoning verification. Did the agent's multi-step reasoning chain contain logical errors? Did it correctly apply a policy to an edge case? These require following an argument and evaluating its coherence.

Nuanced quality assessment. Is this summary good? Does it capture the key points without distorting them? Quality is multidimensional and often requires understanding the source material.

These are real evaluation problems, and LLM-as-Judge is the right tool for them.

The Latency and Cost Argument

Here is the math that most teams do not do before defaulting to LLM-as-Judge for everything.

Running LLM-as-Judge on a single trace: 1-5 seconds latency, $0.01-0.05 in model costs.

Running a heuristic rule on a single trace: <1ms latency, $0.00 in model costs.

At 1,000 traces per day with 4 eval categories each:

Approach	Latency per eval	Cost per eval	Daily cost (4,000 evals)
LLM-as-Judge (all)	1-5 sec	$0.01-0.05	$40-200
Heuristic (all)	<1ms	$0.00	$0
Composite (80/20)	varies	varies	$8-40

At $40-200/day, you are spending $1,200-6,000/month just to evaluate your agent -- before the agent's own inference costs. For a safety check that amounts to "does this string contain an SSN pattern," that spend is hard to justify.

The latency matters too. If evaluation runs inline (before returning the response to the user), adding 1-5 seconds per eval degrades the user experience. Heuristic rules are fast enough to run inline without the user noticing.

The Composite Approach

The right architecture is not heuristic or semantic. It is heuristic for the 80% of evaluations that are pattern-based, and semantic for the 20% that genuinely require language understanding.

The 80% (heuristic): PII detection, prompt injection detection, output completeness checks, cost threshold enforcement, blocklist enforcement, token efficiency, keyword overlap, hallucination marker detection. These run on every trace, inline, in <1ms, for free.

The 20% (semantic): factual accuracy against source documents, nuanced quality scoring, tone assessment, complex reasoning verification. These run selectively -- on a sample of traces, or triggered when heuristic scores fall below a threshold -- and the cost is justified because the evaluation requires actual language understanding.

Iris implements the heuristic side today. LLM-as-Judge is on the roadmap for v0.4.0, and when it ships, it will slot in alongside the heuristic rules as a complementary layer -- not a replacement.

The 12 Built-in Rules

Iris ships with 12 heuristic eval rules across 4 categories. Here is what each category covers and why it does not need an LLM.

Completeness (4 rules)

• non_empty_output -- Output is not empty or whitespace-only. Weight: 2.
• min_output_length -- Output meets a configurable minimum character count (default: 10). Weight: 1.
• sentence_count -- Output contains at least N complete sentences (default: 1). Weight: 0.5.
• expected_coverage -- When an expected output is provided, checks what percentage of key terms appear in the actual output. Passes at 50% coverage. Weight: 1.5.

These are structural checks. An empty response is not a nuance problem. It is a boolean.

Relevance (3 rules)

• keyword_overlap -- Measures word overlap between input and output. If you asked about "password reset" and the response is about "billing," that is detectable without an LLM. Weight: 1.
• no_hallucination_markers -- Flags common AI hedging phrases: "as an AI," "I cannot," "I don't have access." These are exact string matches. Weight: 1.
• topic_consistency -- Measures whether output words relate to input words. A coarse but fast check for topic drift. Weight: 1.

Safety (3 rules)

• no_pii -- Regex patterns for SSN (\d{3}-\d{2}-\d{4}), credit card numbers, phone numbers, and email addresses. Weight: 2.
• no_blocklist_words -- Configurable phrase blocklist. Default includes harmful content patterns. Weight: 2.
• no_injection_patterns -- Regex patterns matching common prompt injection attempts. Weight: 2.

Safety rules have the highest weights because a safety failure matters more than a completeness failure.

Cost (2 rules)

• cost_under_threshold -- Total trace cost must be under a configurable USD threshold (default: $0.10). Weight: 1.
• token_efficiency -- Completion-to-prompt token ratio must be under a configurable maximum (default: 5x). Catches cases where the agent generates disproportionately long responses. Weight: 0.5.

Running an evaluation

{
  "tool": "evaluate_output",
  "arguments": {
    "output": "Navigate to Settings > Security > Reset Password and follow the prompts.",
    "eval_type": "safety",
    "input": "How do I reset my password?"
  }
}

Response:

{
  "score": 1.0,
  "passed": true,
  "rule_results": [
    { "ruleName": "no_pii", "passed": true, "score": 1.0, "message": "No PII detected" },
    { "ruleName": "no_blocklist_words", "passed": true, "score": 1.0, "message": "No blocklisted content found" },
    { "ruleName": "no_injection_patterns", "passed": true, "score": 1.0, "message": "No injection patterns detected" }
  ]
}

Three rules, three results, sub-millisecond. No LLM call, no cost, no non-determinism.

Use the Right Tool for the Check

LLM-as-Judge is a powerful technique. I am building it into Iris because there are evaluations that genuinely require semantic understanding. But the industry's default of routing every evaluation through a judge model is expensive, slow, and unnecessary for the majority of checks agents need.

If you can express the check as a pattern, a threshold, or a string comparison, use a heuristic rule. Save the LLM for the evaluations that actually need one.

Iris is open-source and MIT licensed. The 12 built-in rules are ready to use today. Add it to your MCP config and start evaluating your agent output in <1ms.

npx @iris-eval/mcp-server --dashboard

The code is at github.com/iris-eval/mcp-server. See the roadmap for what is coming next, including LLM-as-Judge in v0.4.0.