AI Agent Tests: Why Refusals Score Like Right Answers
Would you trust an AI agent that never says “I don’t know”?
IN 30 SECONDS
An eval suite is the fixed set of tests a release gate reads, and it scores five things on one run. Refusing when the evidence is thin counts as much as being right, and one successful attack blocks the release.
KEY TAKEAWAYS · 3
- Saying “I don’t know” on thin evidence is scored like accuracy.
- Two equally accurate agents can get different verdicts.
- One successful attack blocks the release, however clean the accuracy run.
On this page
Five things the suite scores
IN SHORT The suite scores five things, and refusing at the right time counts as much as being right.
WORDS TO KNOW
- Grounding
- Every claim in an answer points to a real source document that supports it.
- Abstention
- The agent declining to answer because the evidence is too thin, and handing the case to a person.
- Red-teaming
- Attacking your own agent on purpose, for example with hidden instructions in a document, to find where it breaks.
These are the dimensions I’d make explicit for an agent that reads documents. Their thresholds and coverage belong to the deployment being tested, set before the run with the person who owns the risk, not to this post.
The obvious checks are Grounding Every claim in an answer points to a real source document that supports it. See the glossary and numbers. Abstention The agent declining to answer because the evidence is too thin, and handing the case to a person. See the glossary is easy to forget, and this gate scores it like accuracy.
IN PLAIN WORDS
Abstention is what the verified-citations system produces while the agent runs. Here, the suite checks that it still holds under test.
Coming up next: What happens when two agents are equally accurate?
A worked example: one ship, one hold
IN SHORT Two candidates on the same frozen set: one ships, and the other is held even though its answers are just as accurate.
The values below are invented. The shape is the real one: a candidate, a frozen set, five scores, one verdict.
A WORKED EXAMPLE
Illustrative numbers
PASSCandidate A: a new prompt
INPUT
- Candidate
- agent v1.4, new prompt
- Eval set
- frozen, same as last release
- Includes
- missing-evidence cases
WHAT HAPPENS
- Grounding and numbers: every answer cites a supporting document, every figure matches.
- Abstention: on the missing-evidence cases, it says “I don’t know” and hands them to a person.
- Red-team: no attack gets through. Latency and cost: inside the limits.
OUTPUT
- Verdict
- SHIP, behind human review
HOLDCandidate B: a faster model
INPUT
- Candidate
- agent v1.5, faster model
- Eval set
- the same frozen set
WHAT HAPPENS
- Grounding and numbers: as good as candidate A.
- Abstention: one question needs a lease that is missing. It answers anyway, from a similar lease.
- Red-team: no attack gets through.
OUTPUT
- Verdict
- HOLD
- Next
- tighten abstention, replay the same set
ITEM GET
Refusing right is scored
Coming up next: And what if someone attacks the agent on purpose?
Red-teaming can veto the release
IN SHORT One successful attack blocks the release, even when every accuracy check passed.
Accuracy tells you the agent is right on the cases you thought of. Red-teaming Attacking your own agent on purpose, for example with hidden instructions in a document, to find where it breaks. See the glossary shows what it does on the cases an attacker thought of. In a gate, the second can veto the first.
EXAMPLE
Each attack attempt is a probe. A document tries to overwrite the agent’s instructions: that’s a prompt injection. A question tries to pull data the agent shouldn’t show.
STEP BY STEP
Step 1. Candidate. A new version of the agent enters the gate.
- Candidate A new version of the agent enters the gate.
- Accuracy passes Its answers cite real sources and its numbers match.
- Attack works: blocked A document hides an instruction and the agent follows it. Blocked, whatever the accuracy score.
- Fix, then rerun The hole is closed and the whole suite runs again. If every dimension passes, it ships behind human review, re-evaluated when its data drifts.
Treating red-team findings as known issues to fix later is how an agent ships with a live injection path.
WATCH OUT
Passing a red-team is not the same as containing one. The gate checks resistance. The deployment still needs least privilege so a successful injection cannot move money: the agent gets only the access its job needs.
FAQ
In regulated finance, a confident wrong answer costs more than a refusal. Measure only accuracy and an agent that answers everything looks great, until the case where it should have said “I don’t know.”
STAGE CLEAR
What you now know
- The suite scores sources, numbers, refusals, attack resistance, and speed and cost, on one run.
- Answering without a source fails, even when the answer happens to be right.
- Resisting attacks is not containment: access still has to be limited.
Check yourself
Tap or click a question to see the answer.
Illustrative case: a candidate beats the last release on accuracy, but one attack made it show data the user shouldn’t see. Which verdict?
Block. A successful attack blocks the release, whatever the accuracy. Close the leak, then rerun.
Former CTO of Integra. Named author (1 of 5) of ERC-3643, first author of ERC-6960, co-author of ERC-7410, and co-author of ERC-8203, which is still a draft. Building production AI and regulated Web3 systems.
Have a project to ship or a role to fill?
Discuss your project or roleFollow the engineering notes
Get new articles on production AI and smart contracts by email.