
Our agent platform took 44 seconds to produce the first token on every turn in which the model used tools. The model, the queue, the worker and the guardrails were all fast; we were changing the list of tools on every turn, which invalidated the prompt cache, so each turn paid to write the whole context into the cache and never read it back. The fix was to stop changing the list.
Where the 44 seconds were not
The platform queues its work. A chat turn comes in, is validated and saved, and is handed to a worker that streams the reply back. Guardrails run twice on every turn, once on the way in and once right before the model call. The investigation began as a wider effort to cut time to first token, and the guardrails were the obvious suspect. They took a few milliseconds per pass, both passes, every time.
So we ruled things out in order. The interface rendered at once when bytes arrived. The queue handed the job to a worker in single-digit milliseconds. The worker loaded history and ran its checks in a few milliseconds each. The same prompt, sent to the same model outside our stack, answered in under two seconds. Every layer we owned was fast when measured on its own, and the delay appeared only on turns with tool use. That left one place to look: the request we built and sent.
Why does filtering tools per turn break the prompt cache?
Anthropic’s prompt caching is a prefix cache. The request is serialized as tools, then system prompt, then messages, and what gets cached is a prefix of that sequence. On the next turn you pay full price only for what comes after the unchanged part. For an agent product this is most of the economics, because the prefix is large. A single browser-automation tool server can spend 13,700 tokens describing its tools before the agent does anything, and a heavier one 17,000. Attach a handful of servers and the tools and system prompt run to tens of thousands of tokens before the user has typed a word.
We had been clever about tools. On each turn we computed the tools relevant at that moment and sent only those. For a single request the reasoning holds: fewer tokens spent on descriptions, fewer chances for the model to reach for the wrong tool. For a cached conversation it fails, because the tools come first in the serialized request. Change them and the first bytes change, and nothing after them matches the cache any more.
Our telemetry showed roughly 74,000 cache-write tokens per turn and zero cache reads. The 44 seconds was the cache write. It was also a bill. A cache write is charged at a premium over the base input rate, and we paid that premium on the full prefix, on every tool turn, in conversations that could run for hundreds of turns. By the time we found the bug, it had cost five figures.
Anything that changes the prefix pays for the whole prefix.
The fix, and where we checked it
We froze the serialized tool list. The model now receives the same set every turn and ignores what it does not need, which models are good at. Where a tool really must not be used in a certain state, its description says so, and the server rejects the call deterministically after it is made. The judgment sits with the model or behind a check, never in a prefix rebuilt each turn.
We did not check the plan against documentation or blog posts. We checked it against the installed SDK source, the code that actually serializes the request. Documentation describes intentions; the installed source describes what happens. Reading it confirmed the byte order, tools first, which explained why one changing array spoiled everything behind it. It also turned up a second, smaller offender: a debug timestamp we had left in the system prompt for support tickets. It changed on every request, so it broke the cache even on turns with no tool calls.
The agent that wrote the per-turn filtering had been confident it was correct: cleaner prompts, tighter focus, exactly what the ticket asked for. It was wrong and did not know it. That is the ordinary condition of a self-report, as I describe in Agent self-reports are not evidence.
What we do differently now
After the fix, tool turns went from 44 seconds to normal, with the same model, the same product and the same context. Three practices stayed.
- We measure cache reads and cache writes per turn. Latency told us something was wrong; only the cache counters told us what.
- The prefix is a reviewed surface. A change to the tool list, the system prompt builder or their order gets the scrutiny of a schema migration.
- We track cost per account and per agent. One account writing ten times the cache tokens of the others on the same plan has a broken prefix somewhere in its path.
The last of these would have caught the bug two weeks earlier, and it has since grown into the discipline I describe in Token economics. The check itself is one query against usage data: cache reads per turn. If that number is zero on the second turn of a conversation, something in the prefix is changing.


