SKIP TO CONTENT
AI AGENTS6 minAgents in Production / Ep. 10 · Stage 10-3

AI Agent Test Sets: Keeping the Launch Score Honest

Six months after launch, a release run misses the bar. Did your agent really change, or did the test set?

XIN

IN 30 SECONDS

A frozen eval set is a released version of your test cases, with its hash recorded on every run. When a run misses the bar, the hash tells you whether the set changed.

KEY TAKEAWAYS · 3

  • Change the set only through a new version with a written reason, and record its hash on every run.
  • When a run misses the bar, the agent, the set or the cost behind the threshold may have changed. Finding out which one is most of the work.
  • Put the review date in the file the gate reads: a stale set keeps passing.
On this page

Freeze the set before you measure anything

IN SHORT A frozen set changes only through a new version, with a written reason and a new fingerprint.

WORDS TO KNOW

Frozen eval set
A released version of the eval set that never changes. Any edit makes a new version.
Hash
A short fingerprint computed from the content of the set. Change one case and the fingerprint changes.
Holdout
Cases opened only on release day. Nobody may tune the agent against them.

Adding a case, fixing a wrong label (the expected behavior on a case) or removing sensitive data creates a new version of the Frozen eval set A released version of the eval set that never changes. Any edit makes a new version. See the glossary. Its Hash A short fingerprint computed from the content of the set. Change one case and the fingerprint changes. See the glossary is recorded on every run, so anyone can check that two runs took the same exam.

Keep the old version’s record and the history you are allowed to keep. Never change the comparison set silently to make a candidate pass.

THE SET, IN COMPARTMENTS
Layered stack diagram: 4 layers, top to bottomLayered stack diagram: 4 layers, top to bottom. Layer 1, FROZEN CORE: Real traffic, and the case behind each incident, Labelled synthetic cases; the mix depends on coverage needs, Hashed; any change is a new version. Layer 2, RED-TEAM PACK: Successful attacks, while the threat is still relevant, Versioned; retirements documented. Layer 3, GROWTH POOL: Cases waiting for a label, Not scored until the next freeze. Layer 4, HOLDOUT: Opened only on release day.FROZEN COREReal traffic, and the casebehind each incidentLabelled synthetic cases;the mix depends on coverageneedsHashed; any change is a newversionRED-TEAM PACKSuccessful attacks, whilethe threat is still relevantVersioned; retirementsdocumentedGROWTH POOLCases waiting for a labelNot scored until the nextfreezeHOLDOUTOpened only on release day
Four compartments, four rules.

Thresholds are earned on the frozen core, and nobody is allowed to tune against the Holdout Cases opened only on release day. Nobody may tune the agent against them. See the glossary.

IN PLAIN WORDS

Each case carries the expected behavior: answer, abstain or block. The risk owner sets that label, not the engineer chasing a green run. Have domain reviewers validate the labels and the risk owner approve the acceptance policy.

Coming up next: So what do you check first when a run misses the bar?

When a run misses the bar, find what moved

IN SHORT When a release run comes in under the bar, check what changed before anyone touches the agent.

STEP BY STEP

Flow diagram: 3 stepsFlow diagram: 3 steps. A run misses the bar, then Which of the three moved?. Which of the three moved? branches into 3: If set hash changed, then Comparison void. If set held, agent changed, then Likely regression. If only the cost changed, then Threshold expired. Both paths continue to Write down which one.SET HASH CHANGEDSET HELD, AGENT CHANGEDONLY THE COST CHANGED1A run misses the bar2Which of the three moved?Comparison voidLikely regressionThreshold expired3Write down which one

Step 1. The miss. A release run comes in under the bar. List what changed since the run it is compared to.

Pick a step to see it move
  1. The miss A release run comes in under the bar. List what changed since the run it is compared to.
  2. The set changed The hash differs: you measured a different exam. Re-run on the frozen hash first.
  3. The agent changed Set held, agent changed: likely a regression, the case the gate exists for. Confirm with repeat runs and hold the release.
  4. The cost changed Only the cost behind the bar changed: re-earn it with the risk owner.
  5. Write it down Note which one it was in the run record, next to the hash. That line turns an argument into an audit trail.
One failed run, three diagnoses. Fixing the agent would be wrong in two of them.

The first arm catches people: a set that grew by nine cases is a different exam. A hash on every run makes that a two-minute check, not a two-day argument.

ITEM GET

Check what moved first

Only one diagnosis, a held set with a changed agent, points to a likely regression, a drop on something that used to work.

Coming up next: And what if the set itself has gone stale?

A frozen set has an expiry date

IN SHORT A frozen set doesn’t last forever: it has a lifecycle, and retiring it is a decision.

THE LIFECYCLE
Sequence diagram: 4 participants: Engineer, Frozen set, Release gate, Risk ownerSequence diagram: 4 participants: Engineer, Frozen set, Release gate, Risk owner. Step 1: Engineer to Frozen set, Freezes the cases and records the hash.. Step 2: Engineer to Frozen set, Adds cases or fixes labels in a new version.. Step 3: Frozen set to Risk owner, Signals drift: production has cases the set lacks.. Step 4: Risk owner to Engineer, Orders a new baseline; old bars expire with the old hash.. Step 5: Risk owner to Release gate, Retires the set when its product no longer exists..ENGINEER > FROZEN SETFreezes the cases and recordsthe hash.ENGINEER > FROZEN SETAdds cases or fixes labels ina new version.FROZEN SET > RISK OWNERSignals drift: production hascases the set lacks.RISK OWNER > ENGINEEROrders a new baseline; oldbars expire with the old hash.RISK OWNER > RELEASE GATERetires the set when itsproduct no longer exists.
The lifecycle I propose, one owner per step. The set outlives many releases, not the product.

A stale test set doesn’t start failing. It starts passing, comfortably, on questions your users stopped asking a year ago.

A gate that always says yes costs more than no gate and tells you no more. So put the review date next to the bar, in the file the gate reads.

Asked to evidence accuracy rather than assert it? The bars are part of what you hand over, with the set, the results and the reasoning. That’s the argument of the EU AI Act as an engineering checklist.

FAQ

Enough to cover the important slices and the uncertainty you can tolerate; rare failures need more evidence. One severe failure should still block a zero-failure gate, whatever the set size.

At its review date, and when the task, the risk, the data or the budget changes. A new model needs a fresh evaluation, but doesn’t by itself justify a new acceptance standard. Recheck comparability whenever the set version changes.

STAGE CLEAR

What you now know

  • A frozen set changes only through a new version, with a reason and a new hash.
  • Why a stale set keeps passing, and where the review date belongs.

Check yourself

Tap or click a question to see the answer.

Since the last run, a reviewer fixed three wrong labels in the set. The new score is lower. Is this a regression?

You can’t tell yet. Fixed labels mean a new set version and a new hash, so the two runs took different exams. Re-run both candidates on the same version.

You finished every stage published so far in this series. Back to the series map

Adam Boudjemaa
Adam Boudjemaa

Former CTO of Integra. Named author (1 of 5) of ERC-3643, first author of ERC-6960, co-author of ERC-7410, and co-author of ERC-8203, which is still a draft. Building production AI and regulated Web3 systems.

Have a project to ship or a role to fill?

Discuss your project or role
LinkedInTwitterGitHub
Follow the engineering notes

Get new articles on production AI and smart contracts by email.