SKIP TO CONTENT
HIRING6 minHiring for Production · Episode 1

Hiring an AI Engineer: How to Look Past the Demo

A chat demo answers well on clean sample files. What happens when the wrong user asks, or a connected system goes down?

IN 30 SECONDS

A production AI engineer is someone who builds the software around a model: the links to your systems, the permissions, the tests and the handover. A good demo shows very little of that work. This stage gives you a four-part scorecard.

KEY TAKEAWAYS

· 3
  • Hire for one business task and a measurable test for accepting the work, before you choose an agent framework.
  • Beyond the demo, ask for proof in four areas, access rules included.
  • Access must be checked by the server before the search. A prompt that says “keep this private” is not enough.
On this page

The problem: hiring from the demo

IN SHORT Start from the Workflow One recurring task done in set steps, such as answering a question about a lease. See the glossary the engineer will own, not from the demo.

WORDS TO KNOW

Workflow
One recurring task done in set steps, such as answering a question about a lease.
AI agent
Software where a language model picks some of its own next steps, such as which tool to call.
Retrieval
Searching your documents for the passages most likely to answer one question; the model writes from them (RAG).
Permission boundary
The rule, enforced by the server, that decides what a user or agent may read or do.
Evaluation
A fixed set of test cases with expected results, which can be run before each release.

I delivered a multi-agent platform for a client in private equity (investing in privately held companies) through Ilayer. It plugged into the client’s own business systems. The client stays confidential.

That project shapes how I’d hire for the next one. A polished chat demo tells you very little about permissions, stale source data, or a connected system going down.

STEP BY STEP

Flow diagram: 5 stepsFlow diagram: 5 steps. A user asks a question, then Access checked before search. Access checked before search, then Search your real systems. Search your real systems, then Answer with its source. Answer with its source, then Logged, versioned, can roll back.1A user asks a question2Access checked beforesearch3Search your real systems4Answer with its source5Logged, versioned, canroll back
What this user may read, decided before any search.

Step 1. Ask. A user asks a question about a lease.

Pick a step to see it move
  1. Ask A user asks a question about a lease.
  2. Check access Before any search, the server checks which files this user may open.
  3. Search The search runs on your real systems, which can be stale or time out.
  4. Answer The answer links to its source, or says the evidence is missing.
  5. Trace Logs, versions and a written rollback plan let the next engineer trace a wrong answer and undo a release.
Example: one question’s path in a well-built production system. A chat demo shows little of steps 2, 3 and 5.

Write down the task first. Then what the software may do and the evidence you will accept. A demo is not enough: ask for proof in four areas, as below.

ADAM says: Hire an electrician to wire the office, not to show you a lamp that lights up. Where it breaks: an electrician works to published wiring standards.

AI benchmarks (standardized tests) exist too, but none of them knows your workflow. Your acceptance cases have to come from your own risk.

Anthropic separates workflows (set paths) from AI agent Software where a language model picks some of its own next steps, such as which tool to call. See the glossary (they choose their next steps). Its agent engineering guide advises starting simple. I’d ask which steps need a model at all.

Coming up next: So what do you ask in the interview?

Four areas to test in the interview

IN SHORT Judge four areas (integration, the Permission boundary The rule, enforced by the server, that decides what a user or agent may read or do. See the glossary, evaluation, operations), with a real artifact or a walkthrough for each.

Integration
A source system times out after accepting a request. Then what?
An “unknown outcome” state, a check of what really happened, and no duplicate when a write (a data change) is retried.
Permission boundaries
What stops a user from reading another team’s documents?
A server check before retrieval, plus negative tests (a forbidden request is refused). A prompt (the instruction given to the model) saying “keep this private” is not enough.
Evaluation
Show me a case that should stop a release.
Versioned cases, expected results, a failure report, and a rule to escalate when evidence is missing.
Operations
How does the next engineer investigate a wrong answer?
Traceable sources, model and prompt versions, suitable logs and a written way to roll back.

IN PLAIN WORDS

The walkthrough can use a cleaned-up example. An engineer can keep a non-disclosure agreement (NDA) and still explain their choices.

A tool is not a decision. Amazon Bedrock offers retrieval filters on document labels, such as team or date, for Retrieval Searching your documents for the passages most likely to answer one question; the model writes from them (RAG). See the glossary. Your app still has to decide which filters a signed-in user may ask for.

ITEM GET

Proof, not a demo

In each of the four areas, ask to see the work, not a promise.

For Evaluation A fixed set of test cases with expected results, which can be run before each release. See the glossary, see my evaluation suite that gates a release. Ask which checks belong in plain code and which need judgment.

An overall accuracy score should never hide a permission failure.

Coming up next: What does a weak access check look like?

A worked example: one question, two users

IN SHORT The same question from two users shows why access must be checked before the search.

The names, files and clauses below are invented. They show what to watch for when you switch users.

A WORKED EXAMPLE

Illustrative numbers

PASSAlice, from operations, may read the lease folder

INPUT

User
Alice (operations)
Question
What notice does the Unit 4 lease require?
Files
lease v3 (current), lease v2 (obsolete)

WHAT HAPPENS

  1. The server checks Alice’s access before any search.
  2. The search finds both revisions and keeps v3, the current one.
  3. The answer quotes v3 and links to it.

OUTPUT

Result
answer with a link to lease v3
Check
a reviewer opens the cited clause

LEAK RISKBob, from another team, with a prompt-only build

INPUT

User
Bob (no access to leases)
Question
the same one

WHAT HAPPENS

  1. The build searches every file first.
  2. Then it tells the model to keep other teams’ files private.
  3. The lease is already in the model’s input, and no code stops a quote.

OUTPUT

Result
Bob may see a clause he can’t open
Fix
check access before the search: a clear refusal

PART COMPLETE

What you now know

  • A demo shows the model. Production adds access checks, real systems, tests and a way to roll back.
  • The interview asks for one real artifact or walkthrough per area.
  • Access is checked by the server before the search, never left to the prompt.

Check yourself

Tap or click a question to see the answer.

Illustrative case: a candidate’s evaluation report shows a high overall score, but one failing case let a user read another team’s file. Does the build pass?

No. An overall score should never hide a permission failure. Look for a server check before retrieval, plus negative tests.

Next part:How do you test an AI engineer before you hire, and what do you send them?6 min
Adam Boudjemaa
Adam Boudjemaa

Web3 x AI engineer, former CTO of Integra. Named author (1 of 5) of ERC-3643, first author of ERC-6960, co-author of ERC-7410, and co-author of ERC-8203, which is still a draft.

Have a project to ship or a role to fill?

Discuss your project or role

My work at the web3 x AI intersectionOpen-source tools I build

Follow the engineering notes

Get new articles on production AI and smart contracts by email.