Bank Reconciliation Agent: When It Refuses to Guess
An agent that matches every bank line is easy to build. Would you let it near your fund’s books?
IN 30 SECONDS
Abstention is an agent declining to answer when the evidence is too thin, and handing the case to a person. For this reconciliation agent, that refusal to guess mattered more than its speed.
KEY TAKEAWAYS · 3
- What made the agent safe was abstention: a line it can’t back with evidence goes to a person.
- Retrieval, numeric checks and abstention each catch a different failure, and all of it was evaluated and red-teamed before go-live.
- Zero unsupported answers is a goal of the evaluations, not a guarantee for every answer in production.
On this page
What the agent refuses to match
IN SHORT What makes this agent safe near a fund’s books is that it refuses to guess.
WORDS TO KNOW
- Audit trail
- The records that auditors check later.
- Abstention
- The agent declining to answer because the evidence is too thin. It says “I don’t know” instead of guessing.
- Escalation
- Handing a case to a person, together with what the agent found and why it stopped.
An agent that matches everything is easy to build, but nobody can trust it. One confident wrong match sits in the Audit trail The records that auditors check later. See the glossary and may go unnoticed until the auditors check. Cleaning it up then costs far more than a day.
The goal for this agent was never “match more.” It was “never be confidently wrong.”
The agent can say “I don’t know.” When the evidence doesn’t support a match, the intended path is Abstention The agent declining to answer because the evidence is too thin. It says “I don’t know” instead of guessing. See the glossary, then an Escalation Handing a case to a person, together with what the agent found and why it stopped. See the glossary to a person. The 23-minute result doesn’t show how long each stage took, and it doesn’t guarantee that every future match will be right.
IN PLAIN WORDS
Thin evidence means the agent isn’t sure, so it asks. Wrong numbers mean the evidence contradicts the match, so the match should be blocked outright and a person looks at it.
STEP BY STEP
Step 1. A proposed match. A bank line comes with a candidate document.
- A proposed match A bank line comes with a candidate document.
- Supported The evidence supports it and the numbers agree to the cent: marked as matched, source attached.
- Thin evidence The evidence is thin or missing: the agent abstains, and the line goes to a person with what it found.
- Numbers off The numbers don’t reconcile: the match should be blocked before it can reach a posting step.
More on this rule in the piece on verified citations and abstention.
Coming up next: So what made a fund trust it enough to keep it?
What made it trustworthy enough to keep
IN SHORT Each guardrail catches a different failure, and all of it was evaluated and red-teamed before go-live.
The reconciliation agent shared retrieval, evaluation and guardrails with the wider platform. This post describes the first delivery, which had 11 agents; the platform reached 25 modules at handover. I broke down the full 11-agent platform in its own piece.
Red-teaming means attacking the system on purpose to find its failures. “It worked in the demo” is no standard for a fund’s books.
Coming up next: So if speed isn’t the point, what should you measure?
The real lesson is not the 23 minutes
IN SHORT Speed makes the story worth telling, but mistakes and review decide whether the agent is worth keeping.
To judge another deployment, I’d ask for its starting process, its evaluation failures and its human review load, next to the runtime. Track unsupported answers and wrong matches explicitly.
WATCH OUT
A target of zero on your pre-launch test means little on its own. Say how many cases it had, what they covered, and how you keep watching after launch. Passing it doesn’t prove the system can never be wrong.
Zero unsupported answers is a goal of the evaluations, not a guarantee for every answer in production. The engineering lesson is to make doubt visible and easy to act on.
A team adopting a system like this needs evidence of useful work and a process for mistakes. A speed figure alone can’t show either.
ITEM GET
Make doubt visible
FAQ
No, and that’s deliberate. It proposes matches with their evidence and hands any line it can’t back, or whose numbers don’t add up, to a person. In a regulated fund, a confident wrong match costs far more than a question.
The pattern transfers if your bottleneck is the same: too many sources, too much history, and a review that is really a search problem. Measure your own starting point first, because the 8 hours and 23 minutes belong to this fund, not to yours. Skipping the guardrails doesn’t transfer.
STAGE CLEAR
What you now know
- Why refusing to guess is what makes the agent safe near a fund’s books.
- Thin evidence goes to a person; wrong numbers should block the match.
- What to measure next to speed: wrong matches, unsupported answers and review load.
Check yourself
Tap or click a question to see the answer.
A vendor shows you an agent that matches every bank line and never hands anything to a person. Is that a good sign?
Not on its own. Ask for its wrong matches and what it does when the evidence is thin.
Former CTO of Integra. Named author (1 of 5) of ERC-3643, first author of ERC-6960, co-author of ERC-7410, and co-author of ERC-8203, which is still a draft. Building production AI and regulated Web3 systems.
Have a project to ship or a role to fill?
Discuss your project or roleFollow the engineering notes
Get new articles on production AI and smart contracts by email.