AI Agent Projects: How to Judge One Beyond the Agent Count
One agent cut a reconciliation from 8 hours to 23 minutes, by my own report. What should you ask before trusting it?
IN 30 SECONDS
A forward-deployed engineer is an engineer who works inside a client’s own workflows. Here: the result I report, what it doesn’t prove, and what to ask a builder.
KEY TAKEAWAYS · 3
- One of the 11 agents cut bank reconciliation from 8 hours to 23 minutes, a result I report myself, not an independent audit.
- Forward-deployed means direct access to the people who own the workflow, even when the work is remote.
- To judge an AI build, ask for the integration boundary, a failed test case and the handover, not the agent count.
On this page
One agent, unpacked: 8 hours to 23 minutes
IN SHORT The figure below is first-party: I report it myself from this engagement, and it is not an independent audit.
WORDS TO KNOW
- Bank reconciliation
- Checking every line of a bank statement against the fund’s own books, and explaining each gap.
- Baseline
- How long the work took before the tool, measured the same way, so the after figure means something.
- Forward-deployed engineer
- An engineer who works inside a client’s workflow, from the first problem to delivery, on site or remotely.
First-party result for this engagement
First-party result for this engagement
Bank reconciliation Checking every line of a bank statement against the fund’s own books, and explaining each gap. See the glossary means checking each line of the bank statement against the fund’s own books. Search, number checks and a person’s review all fed it.
WATCH OUT
The 23 minutes doesn’t show how long each stage took. It shouldn’t be generalized to another fund without a new Baseline How long the work took before the tool, measured the same way, so the after figure means something. See the glossary: that fund’s own before measurement.
That agent has its own episode.
Coming up next: So how does a remote engineer get that close to the work?
What forward-deployed meant here
IN SHORT Being forward-deployed means working close to the client’s workflow, wherever your desk is.
A Forward-deployed engineer An engineer who works inside a client’s workflow, from the first problem to delivery, on site or remotely. See the glossary follows one client workflow, from the user’s problem to delivery and feedback. I did it remotely, and fit the platform into the fund’s own systems.
What matters is direct access to the people who own the workflow. Remotely, that can mean scheduled walkthroughs, shared examples and short feedback cycles.
STEP BY STEP
Step 1. See the real work. The analyst shows a real search, as it’s really done.
- See the real work The analyst shows a real search, as it’s really done.
- Change it I change the implementation to fit that workflow.
- Check it The analyst checks the changed path against the original problem.
IN PLAIN WORDS
The engineer learns the real work, then checks each change with the person who does it.
Coming up next: So what should you ask before you hire someone for this?
What to ask before you hire this
IN SHORT Agent count describes the build. It isn’t a quality score.
Ask any AI builder for three things instead:
- the integration boundary: where it plugs into your tools, and where it stops;
- a failed evaluation case;
- the handover: who runs it once the builder leaves.
The consultancy slide
The embedded engineer
ITEM GET
Ask for the boundary, a failure, the handover
To tell a real deployment from a demo, I use the FDE Evidence Ladder.
FAQ
Yes, alone and remotely, through Ilayer. The shared base is what let one person build them all, though each new agent still needed its own tests and permissions.
It’s their data and their business, not a logo for my portfolio. I can share the work, the data scale and the design choices. I won’t name the client or invent per-agent numbers.
It transfers when your bottleneck is the same: too many sources, too much history, and reviews that are really searches. Skipping the guardrails doesn’t transfer. Search, number checks and the rule to abstain come first.
STAGE CLEAR
What you now know
- 8 hours to 23 minutes is a result I report myself; another fund needs its own baseline.
- Forward-deployed work means direct access to the people who own the workflow, even remotely.
- Agent count describes a build; the boundary, a failed test and the handover show its quality.
Check yourself
Tap or click a question to see the answer.
A vendor pitches dozens of AI agents, and its demo never says “I don’t know”. Does either fact show quality? What should you ask for?
Neither. The count only describes the build, and a demo that never abstains doesn’t show how it handles thin evidence. Ask for a failed test case, the integration boundary and the handover.
Former CTO of Integra. Named author (1 of 5) of ERC-3643, first author of ERC-6960, co-author of ERC-7410, and co-author of ERC-8203, which is still a draft. Building production AI and regulated Web3 systems.
Have a project to ship or a role to fill?
Discuss your project or roleFollow the engineering notes
Get new articles on production AI and smart contracts by email.