Shipping an 11-agent GenAI platform, solo, into regulated finance

A regulated financial organization had years of documents across many systems and needed answers it could actually trust: grounded in the source, numerically correct, and safe to act on. The job was to connect the archive to a working retrieval system, verify its outputs and give the receiving team a system it could operate.

The corpus was large and messy: I ingested roughly 2 TB and 1.4 million files spanning 13 years from 13 sources. That scale is the point, because a system that works on a clean sample and falls over on the real archive is exactly the failure this work exists to prevent.

Real regulated-finance constraints shaped every decision: scoped and audited data access, a low tolerance for wrong answers, and the need for the system to be explainable after the fact. The stakeholders were the people who would rely on the output day to day, so the bar was operational trust, not a benchmark score.

A RAG backend on FastAPI with Postgres and pgvector, an ingestion layer using LLM-vision OCR with a Textract fallback, and a team of 11 orchestrated agents over retrieval.

The part that mattered most was the verification layer: byte-level verified citations (every claim opens to the exact source span), NLI faithfulness checks (a model checks whether the source span supports the claim; it can be wrong), numeric-consistency checks (every figure reconciles with the source), and an abstention path so the system declines when the evidence is thin instead of guessing.

I rejected the alternative that most demos take, a single retrieval-and-answer call with a "sources" footer, because in a regulated setting an unverifiable answer is worse than no answer. The verification stack is the reason this reached production.

The system was covered by an evaluation suite plus red-teaming, with a gate designed to catch regressions in the tested cases. Passing those tests does not cover every possible production failure. Known failure cases informed the regression tests. The failure modes worth designing against were the usual ones for this work: confident wrong answers, numbers that looked right but did not reconcile, and retrieval that fell apart on the genuinely messy parts of the corpus.

Human-in-the-loop and escalation were part of the design, not an afterthought: when the system abstained, a person picked it up. Operations included observabilityover what the agents did, and cost caps with model routing so the economics stayed sane at production volume. Forward-deployed means I stayed with it until it could be operated, not just until it demoed.

MEASUREDThe platform was in production during the engagement. One measured workflow result: a bank-reconciliation workflow went from about 8 hours to roughly 23 minutes. This is my record of that workflow, not an independently audited benchmark or a promise for another deployment. Source-linked answers and explicit abstention were part of the delivery design.

This deployment is where the FDE Evidence Ladder comes from. It informed the verification sequence, Rungs 2 to 5 (verified citation, faithfulness, numeric consistency, abstention), and stood on the standard rungs beneath and above it: grounded retrieval, evaluation and red-teaming, and governed operation.

My main lesson was to develop the verification and evaluation paths alongside retrieval. I would build the evaluation harness earlier in the next engagement so failures are easier to reproduce before the system reaches more users.