SKIP TO CONTENT
AI AGENTS6 minAgents in Production / Ep. 10 · Stage 10-2

AI Agent Mistakes: How Their Cost Sets the Launch Score

In a worked example, candidate A fails 14 test cases and candidate B only 10. Yet B is the one that gets blocked.

XIN

IN 30 SECONDS

A launch score is set per kind of mistake: a zero bar where a wrong answer has no nameable price, a budget where the cost is reviewer minutes.

KEY TAKEAWAYS · 3

  • Price both ways of being wrong. Block known severe failures in the eval set, but zero observed failures does not prove zero production risk.
  • Set review-rate limits from a documented review capacity: the escalation budget is the minutes you agree to pay for.
  • Fewer failures can still block a release, if one lands in the class whose bar is zero.
On this page

Price both directions of being wrong

IN SHORT Being wrong comes in directions with very different prices, so each direction gets its own kind of limit.

WORDS TO KNOW

Threshold
The score a new version must reach on one measured dimension before it may ship. Also called the bar.
Thin evidence
A case where the documents don’t really support an answer, so the agent should pass it to a person.
Escalation
The agent hands a case to a person instead of answering it.

The costly direction is answering on Thin evidence A case where the documents don’t really support an answer, so the agent should pass it to a person. See the glossary: possibly nobody notices until an audit. It’s what verified citations and abstention are designed to prevent while the agent runs. Here it gets a Threshold The score a new version must reach on one measured dimension before it may ship. Also called the bar. See the glossary instead: zero, because nobody can name its price.

STEP BY STEP

Decide: Which direction of wrong is expensive in your domain?Decision tree: Which direction of wrong is expensive in your domain? If a confident wrong answer reaches someone who will act on it, then The bar is zero on that class. If the expensive direction is an unnecessary refusal, then Give it a budget, not a bar. If both directions cost you almost nothing, then You don’t need a gate here.DECIDEWhich direction of wrong isexpensive in your domain?If a confident wrong answerreaches someone who willact on itThe bar is zero onthat classFor this example, anyobserved severe failureblocks release. Passingonly describes the testedcases: bound real-worldauthority, keep runtimecontrols and monitor theremaining risk.If the expensive directionis an unnecessary refusalGive it a budget, nota barDecide what the queue canabsorb each month and letthe percentage follow.If both directions cost youalmost nothingYou don’t need a gatehereUse a small regressionsuite, basic safety checksand production monitoring.

Step 1. The question. Which way of being wrong is expensive in your domain?

Pick a step to see it move
  1. The question Which way of being wrong is expensive in your domain?
  2. Zero bar A confident wrong answer would reach someone who acts on it: blocked.
  3. A budget Price the reviewer minutes of each unnecessary refusal, and renew the budget.
  4. No gate Cheap both ways: spend the deeper evaluation budget where failures cost more.
Match the strength of the gate to the severity of failure. Block known severe failures; use proportionate regression checks and monitoring for low-impact features.

A slow agent gets a ceiling set by the review window. A leak gets no threshold at all: it blocks the release.

WATCH OUT

A read-only drafting tool and an agent that can move money need different evidence. Both benefit from regression checks. Broader authority warrants stronger tests, explicit approvals and tighter operational controls.

Coming up next: So what happens when two candidates meet these two bars?

A worked example: two candidates, one blocked

IN SHORT On the same illustrative set, the candidate with fewer failures is the one that gets blocked.

Every value here is invented: the arithmetic transfers, the values never do. The set has 200 cases: 160 the agent can answer, and 40 with thin evidence, where it should escalate.

IN PLAIN WORDS

An unnecessary Escalation The agent hands a case to a person instead of answering it. See the glossary is a case the agent passes to a person when it could have answered. Each one costs reviewer time, so it has a price.

Two bars apply: zero thin-evidence cases answered, and at most 10% unnecessary escalations, 16 of 160.

A WORKED EXAMPLE

Illustrative numbers

PASSCandidate A clears both bars

INPUT

Thin evidence, answered anyway
0 of 40
Unnecessary escalations
14 of 160
Total failures
14

WHAT HAPPENS

  1. Zero bar on thin evidence: none observed, so it holds.
  2. 14 of 160 is 8.75%, under the 10% budget.
  3. At about $3 each, these escalations cost about $42 a run.

OUTPUT

Result
can ship

BLOCKEDCandidate B: fewer failures, still blocked

INPUT

Thin evidence, answered anyway
1 of 40
Unnecessary escalations
9 of 160
Total failures
10

WHAT HAPPENS

  1. 9 of 160 is about 5.6%, well under the budget.
  2. But one thin-evidence case got a confident answer, and that bar is zero.
  3. B has fewer failures than A. The class that matters failed.

OUTPUT

Result
blocked
Next
back into the loop

The cost per escalation is the only input with a source: about three reviewer minutes at a modelled $60 an hour, from what it costs to run 46 agents. That is about $3 each.

$48

a run for 16 unnecessary escalations

Illustrative: 16 × about $3, from episode 7’s modelled assumptions.

16 is the count this queue agreed to absorb. So the bar isn’t “10 percent accuracy”. It reads: we will pay for 16 of these, and here is the rate that produced that count.

Change one input and the bar moves: halve the reviewer rate, or move review from same-day to two hours. The number should move when your cost of being wrong moves.

ITEM GET

Fewer failures can still block

Candidate B failed less often than A, but one failure landed in the class whose bar is zero.

WATCH OUT

The zero bar blocks any observed failure in its severe class. That is an acceptance decision, not a statistical claim that the true failure rate is zero. If the system can’t meet it, reduce its authority or change the workflow. Don’t treat the test set as proof of safety.

FAQ

Agree on the unacceptable failures and the review budget with the workflow owners, and measure on a versioned, representative set. A zero-failure gate can veto known severe failures; it can’t prove unseen ones are impossible. Derive the other limits from capacity, quality needs and uncertainty, and document why.

STAGE CLEAR

What you now know

  • Why each way of being wrong gets its own limit: a zero bar, a budget or a block.
  • Why the candidate with fewer failures can still be the one that’s blocked.

Check yourself

Tap or click a question to see the answer.

In the worked example, the review team doubles in size. Which bar could be re-priced, and which one should not move?

The escalation budget can be re-priced with the risk owner: it comes from what the queue can absorb. The zero bar on thin evidence stays: it was never priced in minutes.

Next stage:How do you keep the test set and its threshold honest as the months go by?6 min
Adam Boudjemaa
Adam Boudjemaa

Former CTO of Integra. Named author (1 of 5) of ERC-3643, first author of ERC-6960, co-author of ERC-7410, and co-author of ERC-8203, which is still a draft. Building production AI and regulated Web3 systems.

Have a project to ship or a role to fill?

Discuss your project or role
LinkedInTwitterGitHub
Follow the engineering notes

Get new articles on production AI and smart contracts by email.