Every LLM tool call re-sends your entire conversation
When it comes to LLM chats, I know long context costs more, so I try to keep mine short. What I couldn’t explain was why a single prompt still billed for many times more tokens than the context window holds.
Every time an AI agent calls a tool, it re-sends the entire conversation. The Messages API is stateless, so the model remembers nothing between requests. When it calls a tool, the request ends, your client runs the tool, appends the result to the history, and POSTs the whole thing back as a new request. You pay input tokens to remind it every time.
So a chat near the 1M-token context limit that makes 20 tool calls pays for that 1M about 20 times over, turning it into 20M billed tokens and around $20 on today’s frontier models. That is 20 times the context window, from a single prompt.
Prompt caching helps keep the cost down. Normally the model reprocesses the whole conversation on every request, but a cache lets it reuse the unchanged prefix, so re-sending that part costs about a tenth of full price. You pay a small premium to store it the first time, then read it back cheap after, the same way OpenAI (automatic caching) and Gemini (implicit caching) do it. The reuse only holds while the start of the prompt stays byte-for-byte identical, so a changing timestamp in your system prompt breaks the cache and puts you back at full price.