Shipping an 11-agent GenAI platform, solo, into regulated finance

A regulated financial organization had years of documents across many systems and needed answers it could actually trust: grounded in the source, numerically correct, and safe to act on. The pilots that get demoed in this space usually die on contact with the real, messy corpus. The job was to cross that gap and put something into production that the team would still be using and trusting months later.

The corpus was large and messy: I ingested roughly 2 TB and 1.4 million files spanning 13 years from 13 sources. That scale is the point, because a system that works on a clean sample and falls over on the real archive is exactly the failure this work exists to prevent.

Real regulated-finance constraints shaped every decision: scoped and audited data access, a low tolerance for wrong answers, and the need for the system to be explainable after the fact. The stakeholders were the people who would rely on the output day to day, so the bar was operational trust, not a benchmark score.

A RAG backend on FastAPI with Postgres and pgvector, an ingestion layer using LLM-vision OCR with a Textract fallback, and a team of 11 orchestrated agents over retrieval.

The part that mattered most was the verification layer: byte-level verified citations (every claim opens to the exact source span), NLI faithfulness checks (the claim has to be entailed by the span, not just near it), numeric-consistency checks (every figure reconciles with the source), and an abstention path so the system declines when the evidence is thin instead of guessing.

I rejected the alternative that most demos take, a single retrieval-and-answer call with a "sources" footer, because in a regulated setting an unverifiable answer is worse than no answer. The verification stack is the reason this reached production.

The system was covered by an evaluation suite plus red-teaming, with a regression gate so changes could not quietly make it worse. Every failure I found became a new eval. The failure modes worth designing against were the usual ones for this work: confident wrong answers, numbers that looked right but did not reconcile, and retrieval that fell apart on the genuinely messy parts of the corpus.

Human-in-the-loop and escalation were part of the design, not an afterthought: when the system abstained, a person picked it up. Operations included observabilityover what the agents did, and cost caps with model routing so the economics stayed sane at production volume. Forward-deployed means I stayed with it until it could be operated, not just until it demoed.

MEASUREDThe platform reached production and stayed there. One concrete, defensible number: a bank-reconciliation workflow went from about 8 hours to roughly 23 minutes. The win that matters more is qualitative: answers the team could trust because every one of them could be opened, checked, and, when the evidence wasn't there, declined.

This deployment is where the FDE Evidence Ladder comes from. It climbed the distinctive spine, Rungs 2 to 5 (verified citation, faithfulness, numeric consistency, abstention), and stood on the standard rungs beneath and above it: grounded retrieval, evaluation and red-teaming, and governed operation.

The lesson that generalizes: in regulated finance, the verification layer is not a feature you add at the end, it is the reason the system is allowed to exist. Build it first. What I would do differently is invest even earlier in the eval harness, because every hour spent there paid for itself in trust later.