Tracing AI Agents: The Spans of One Run and Its Tokens
A cost report shows 17,200 input tokens for a call that read 9,200 (made-up numbers). Where did the extra 8,000 come from?
IN 30 SECONDS
A multi-agent trace is the tree of spans of one run. In a coordinated run: a workflow span, one span per agent invocation, and the model and tool calls under each. On each model call, gen_ai.usage.input_tokens should already include cached tokens, so adding the cache fields on top counts them twice.
KEY TAKEAWAYS · 3
- A coordinated multi-agent run can be one trace: an invoke_workflow span, one invoke_agent span per agent invocation, and inference and execute_tool spans under each.
- gen_ai.agent.version is no longer on the in-process agent span (#322). If an auditor will ask which version acted, carry it yourself.
- gen_ai.usage.input_tokens should already include cached tokens. Price the cached part at its own rate; never add it on top.
On this page
One span per agent invocation, its calls nested below
IN SHORT In a coordinated run, one workflow span holds one span per agent invocation, and each one’s model and tool calls nest underneath.
WORDS TO KNOW
- Span
- One timed step of work, such as a model call or a tool call, with a name and labelled values called attributes.
- Trace
- All the spans of one run, linked parent to child, so you can see which step started which.
- Token
- A small piece of text, often part of a word. Models read and write tokens, and providers often bill by them.
- Cached tokens
- Input a provider already processed on an earlier call and reuses, often billed at a different rate.
The agent spans page names each level (no release yet). A coordinated run can be wrapped in a Span One timed step of work, such as a model call or a tool call, with a name and labelled values called attributes. See the glossary named invoke_workflow. Each agent invocation inside it gets invoke_agent, followed by its name.
IN PLAIN WORDS
A span’s kind says whether the work stays in your process or goes out to another service. The conventions recommend INTERNAL for an agent running in your own process, and CLIENT when you call a remote or hosted agent.
ONE AGENT, ONE RUN
Step 1. The agent starts. The workflow span opens an invoke_agent span for this agent.
- The agent starts The workflow span opens an invoke_agent span for this agent.
- It calls the model The call becomes an inference span named after the operation and the model. Tokens and the response id live here.
- The model asks for a tool That answer belongs to the inference span; no new span opens yet.
- The tool runs The agent runs it inside an execute_tool span, with the tool’s name and call id.
- A second model call The tool result goes back to the model in a second inference span, again a child of the agent span.
- The agent ends Its span closes and the next agent opens its own, in the same trace.
WATCH OUT
gen_ai.agent.id should hold the stable id a provider gives a hosted agent. The spec marks in-memory instance ids there as NOT RECOMMENDED. The attribute was also removed from the in-process agent span.
ADAM says: gen_ai.agent.version is gone from that in-process span too (#322). An auditor will ask which version of which agent did this, so carry it yourself.
There are agent-level metrics too: gen_ai.invoke_agent.duration, .inference_calls and .tool_calls. All three are histograms: they record how values are spread.
The inference_calls and tool_calls counts are the cheapest early warning I know of for an agent that got chattier after a prompt change.
A worked example: one model call, counted right
IN SHORT On a model call, gen_ai.usage.input_tokens should already include the cached tokens, so adding the cache fields on top counts them twice.
The Anthropic page spells it out: Anthropic’s own input_tokens excludes Cached tokens Input a provider already processed on an earlier call and reuses, often billed at a different rate. See the glossary. So the conventional value is input_tokens + cache_read_input_tokens + cache_write_input_tokens. The values below are invented.
A WORKED EXAMPLE
Illustrative numbers
PASSOne reconciliation step, counted by the conventions
INPUT
- Uncached input
- 1,200 tokens
- Read from cache
- 8,000 tokens
- Written to cache
- 0
WHAT HAPPENS
- The instrumentation adds the three: 1,200 + 8,000 + 0.
- It puts 9,200 on gen_ai.usage.input_tokens, and 8,000 on gen_ai.usage.cache_read.input_tokens.
- Your cost report prices the 8,000 at the cache-read rate, any cache writes at the cache-write rate, and the 1,200 at the full rate.
OUTPUT
- Input tokens on the span
- 9,200
- Billed
- each part at its own rate
COUNTED TWICESame span, summed the naive way
INPUT
- gen_ai.usage.input_tokens
- 9,200
- gen_ai.usage.cache_read.input_tokens
- 8,000
WHAT HAPPENS
- A report adds the cache field on top of input_tokens, as if they were separate.
- 9,200 + 8,000 gives 17,200, almost twice the real input.
- Priced at the full input rate, the cached 8,000 costs far more than it should.
OUTPUT
- Input tokens in the report
- 17,200
- Result
- cost overstated
ITEM GET
Count input once
Names, kinds and keys follow the conventions at commit 4f85037. com.acme.* stands for your own reverse domain, as OpenTelemetry’s naming guide advises. Values are made up and flattened.
{
"spans": [
{
"name": "invoke_agent recon-agent",
"kind": "INTERNAL",
"attributes": {
"gen_ai.agent.name": "recon-agent",
"com.acme.agent.version": "4.2.0",
"com.acme.audit.record_id": "01K4R7QW3M8ZC2VJ9F0X6TBNPD"
}
},
{
"name": "chat claude-haiku-4-5-20251001",
"kind": "CLIENT",
"attributes": {
"gen_ai.provider.name": "anthropic",
"gen_ai.response.model": "claude-haiku-4-5-20251001",
"gen_ai.usage.input_tokens": 9200,
"gen_ai.usage.cache_read.input_tokens": 8000,
"gen_ai.usage.output_tokens": 310
}
}
]
}There’s no gen_ai.conversation.id here. When none exists, the spec says instrumentations SHOULD NOT make one up from a generated id or a trace id.
Sources, read on 7 October 2026: agent spans, spans, client inference, Anthropic.
FAQ
Wrap the run in an invoke_workflow span; each agent invocation gets an invoke_agent span. Model calls nest below as inference spans, and tool calls as execute_tool spans. Trace context does the parenting.
STAGE CLEAR
What you now know
- How the workflow, agent, model and tool spans of one run can nest in a single trace.
- Why adding the cache fields to gen_ai.usage.input_tokens can count cached tokens twice.
Check yourself
Tap or click a question to see the answer.
A made-up span shows input_tokens 5,000, cache_read.input_tokens 4,000 and no cache writes. How many input tokens were not cached?
1,000. Per the conventions, input_tokens already includes the 4,000 cached tokens, so you subtract them, never add them.
Web3 x AI engineer, former CTO of Integra. Named author (1 of 5) of ERC-3643, first author of ERC-6960, co-author of ERC-7410, and co-author of ERC-8203, which is still a draft.
Have a project to ship or a role to fill?
Discuss your project or roleMy work at the web3 x AI intersectionOpen-source tools I build
Follow the engineering notes
Get new articles on production AI and smart contracts by email.