AI Citation Checks: How Meaning and Numbers Get Tested
Your AI writes “about 60 days” and cites a clause that says 90. Which check stops it?
IN 30 SECONDS
A citation check is a test that the cited passage supports the sentence and that its figures match the source. If either test fails, or the evidence is missing or in conflict, the system abstains.
KEY TAKEAWAYS · 3
- A verified citation matches the stored source exactly. The passage must support the sentence and, when the claim has figures, every figure must match. A pass supports the answer; it doesn’t prove the source or the answer true.
- A neutral passage counts as unsupported. In this design, one failing claim stops the whole answer.
- The system refuses on evidence states written down in advance: no source, sources that disagree, or a number that doesn’t match.
On this page
How you verify a citation, not just attach one
IN SHORT Attaching a citation is easy. Verifying it means an exact match with the stored source, a meaning check and, when needed, a number check.
WORDS TO KNOW
- Source span
- The exact passage of a source document that one sentence of the answer relies on.
- Entailment check (NLI)
- A model test of whether a passage supports a sentence, contradicts it, or does neither. NLI stands for natural language inference.
- Abstention
- The system declining to answer because the evidence is thin. It says “I don’t know” instead of guessing.
- Escalation
- Handing the question to a person, with what the system found and where it stopped.
EXAMPLE
A draft cites the right lease clause but says 60 days where it says 90. Another cites a real page that never makes the point. Both look sourced, but the first contradicts its source and the second is unsupported.
For meaning, the system runs an Entailment check (NLI) A model test of whether a passage supports a sentence, contradicts it, or does neither. NLI stands for natural language inference. See the glossary, or NLI. Does the cited Source span The exact passage of a source document that one sentence of the answer relies on. See the glossary support the claim, contradict it, or neither? Neither isn’t good enough: neutral means unsupported, and unsupported leads to abstention.
Then come the numbers. Language models write figures fluently and still get them wrong. An amount, a date or a count must match the source exactly.
In finance, “about right” is how you book a wrong number with confidence.
STEP BY STEP
Step 1. Exact match. First, the cited passage must match the stored document exactly.
- Exact match First, the cited passage must match the stored document exactly.
- Meaning check Does the passage support the sentence, contradict it, or neither?
- Number check When the sentence has figures, each must match the passage to the digit.
- Supported Every required check passes: the claim can go out with its citation.
- Contradicted The passage refutes the claim, or a figure differs: the answer is blocked and the mismatch flagged.
- Neutral The passage neither supports nor refutes: the system abstains and escalates to a person.
In this design, one unsupported claim that the checks catch makes the whole answer abstain. Sending only the good parts would need a written partial-answer policy, with its own evaluation cases.
WATCH OUT
A pass means support, not proof: the checker is a model and can be wrong, so it must be tested too. The source may also be out of date or not authoritative. And abstention only routes the failures the checks detect.
Coming up next: What does that look like on a lease?
A worked example: one answer sent, one blocked
IN SHORT One question, two drafts: the first passes both checks, the second changes a number and gets blocked.
The lease, the clause and the numbers below are invented.
A WORKED EXAMPLE
Illustrative numbers
PASSDraft 1: the number matches
INPUT
- Question
- How much notice does the Unit 4B lease require?
- Source span
- Clause 12.1: “Tenant shall give 90 days’ written notice.”
- Draft
- The lease requires 90 days’ written notice.
WHAT HAPPENS
- The draft cites clause 12.1.
- Meaning check: the passage supports the sentence.
- Number check: 90 in the draft, 90 in the passage.
OUTPUT
- Verdict
- supported
- Result
- answer sent, clause 12.1 cited
BLOCKEDDraft 2: the number drifts
INPUT
- Source span
- The same clause 12.1
- Draft
- The lease requires about 60 days’ notice.
WHAT HAPPENS
- The draft cites the same clause, so it looks sourced.
- The number check finds 60 in the draft and 90 in the passage.
- The verdict is contradicted: the answer is blocked and the mismatch flagged.
OUTPUT
- Verdict
- contradicted
- Result
- blocked, mismatch flagged
ITEM GET
Close is not a match
Coming up next: And when no passage fits, or two of them disagree?
When the system is allowed to refuse
IN SHORT The system refuses when the evidence is in a state written down in advance, never because it feels unsure.
The table compares it with a typical vendor bot, an assistant supplied by an outside company. It covers the failures the checks detect, with no guarantee of catching every one.
IN PLAIN WORDS
An Abstention The system declining to answer because the evidence is thin. It says “I don’t know” instead of guessing. See the glossary is not a dead end. With an Escalation Handing the question to a person, with what the system found and where it stopped. See the glossary, a person gets the question, the sources found and the point where the check stopped.
STAGE CLEAR
What you now know
- What the meaning check and the number check each look for.
- Why a neutral passage counts as a fail, not a pass.
- Which evidence states lead to a refusal, decided before launch.
Check yourself
Tap or click a question to see the answer.
Illustrative case: an answer has three claims. Two are supported. The third gets a neutral verdict. What does the system return?
It abstains on the whole answer and escalates. Neutral means insufficient, and one failing claim stops everything. Sending the two good claims would need an explicit partial-answer policy.
Illustrative case: two retrieved leases give different notice periods for the same unit. Both citations are real. What does the system do?
It abstains and escalates the conflict to a person, without picking the likelier source: a conflict is an evidence state with its own branch.
Former CTO of Integra. Named author (1 of 5) of ERC-3643, first author of ERC-6960, co-author of ERC-7410, and co-author of ERC-8203, which is still a draft. Building production AI and regulated Web3 systems.
Have a project to ship or a role to fill?
Discuss your project or roleFollow the engineering notes
Get new articles on production AI and smart contracts by email.