Prompt caching for agent loops: what it changes about cost per run

A pull-request review agent resends its whole growing history on every tool call, so the prompt gets bigger every step: by the last call in a 12-file PR, it can be several times the size of the first. Priced without caching, at $2 per million input tokens and $10 per million output tokens, a 14-step run like that costs about $0.29 total, almost all of it the plain input rate billed over and over on that growing prefix. Turn prompt caching on, and the bill drops, but not evenly: the response's usage object used to hand back one input-token count; now it hands back three numbers, and only one of them is cheap.

This post splits every step into fresh, cache-write and cache-read tokens, recomputes the same run's cost with caching on, and shows the one way caching can quietly cost you more instead of less.

Three ways an input token gets billed

Every call in the loop now carries a cache breakpoint, a marker in the request that tells the provider "cache everything up to here." What comes back on the response's usage object splits input into three counts instead of one.

Fresh tokens bill at the plain input rate: no discount and no premium. Cache-write tokens are the ones this call is caching for the first time, because the marked prefix, or part of it, didn't match anything already stored; the provider does the caching work now, so it charges a premium over the base rate. Cache-read tokens are the ones that matched a prefix an earlier call in this same run already cached; the provider is replaying stored state instead of reprocessing it, so it's billed at a steep discount off the base rate.

A cache hit is a call where some, or all, of the marked prefix matches what's already cached; a cache miss is one where it doesn't, whether nothing has been written yet or the cached copy's cache lifetime (sometimes called a TTL, time-to-live: how long the provider holds a written entry before it expires) has passed.

Pull the three counts off the response with a single read_usage function that returns four numbers, fresh, cache-write, cache-read and output, instead of two.

Where the breakpoint goes, and why it has to move

There's only one breakpoint in this loop, and it goes at the end of the message list, on every call. Each step's prefix is the entire previous step's payload: the system prompt, the tool list, the PR description, and every earlier tool result, exactly what run_agent already builds up in messages. Marking "everything so far" means moving the marker forward on every single call; a breakpoint left at step 3 caches nothing past step 3, and step 14 pays fresh or write prices for the ten steps of history added since.

The first call in a run has nothing to read yet, so its whole prefix is a cache-write: 1,700 tokens, written and waiting. The second call's prefix matches that write exactly, so those 1,700 tokens come back as a cache-read, and only the roughly 1,000 tokens the second call adds, the first tool result, go out as a fresh write. From here the pattern repeats: read everything the previous call wrote, write only the delta.

Extending metered_call to handle caching is a matter of marking the breakpoint before the call and reading four numbers instead of two:

CACHE_WRITE_PER_M = 2.50   # placeholder: USD per million cache-write tokens
CACHE_READ_PER_M = 0.20    # placeholder: USD per million cache-read tokens

def read_usage(response):
    """The only provider-specific function. Swap it for another SDK."""
    usage = response.usage
    return (
        usage.input_tokens,
        usage.cache_creation_input_tokens,
        usage.cache_read_input_tokens,
        usage.output_tokens,
    )

def with_cache_breakpoint(messages):
    """Mark the last block of the last message as an ephemeral cache breakpoint, so
    the whole prefix up to here is what gets cached. Call this fresh every step:
    the marker has to move forward, or you're caching a prefix nobody sends again."""
    marked = [dict(m) for m in messages]
    blocks = marked[-1]["content"]
    if isinstance(blocks, str):
        blocks = [{"type": "text", "text": blocks}]
    blocks = [dict(b) for b in blocks]
    blocks[-1]["cache_control"] = {"type": "ephemeral"}
    marked[-1]["content"] = blocks
    return marked

def metered_call(review_id, **request):
    request["messages"] = with_cache_breakpoint(request["messages"])
    response = client.messages.create(model=MODEL, **request)
    fresh, cache_write, cache_read, tokens_out = read_usage(response)
    dollars = (
        fresh * INPUT_PER_M
        + cache_write * CACHE_WRITE_PER_M
        + cache_read * CACHE_READ_PER_M
        + tokens_out * OUTPUT_PER_M
    ) / 1_000_000
    step = sum(1 for row in log if row[0] == review_id) + 1
    log.append((review_id, step, fresh, cache_write, cache_read, tokens_out, dollars))
    return response

Marking the last block of the last message is the whole trick: it tells the provider "cache the prefix through here," and since the message list only ever grows, "through here" always means "everything so far." Check your provider's minimum cacheable prefix length too: a very short first call, well under whatever floor your provider sets, might not be cached at all, no matter where you put the marker.

Recompute the run

Same run, same 14 steps, same placeholder base and output prices as above ($2 per million input tokens, $10 per million output tokens), plus two new placeholder prices for this post: $2.50 per million for cache-write tokens, a 25% premium over base, and $0.20 per million for cache-read tokens, a 90% discount off base.

Step	Tool called	Cache-write	Cache-read	Output	Dollars	Cumulative
1	`read_file`	1,700	0	120	0.0055	0.0055
2	`read_file`	1,000	1,700	80	0.0036	0.0091
3	`read_file`	1,100	2,700	80	0.0041	0.0132
4	`read_file`	1,000	3,800	80	0.0041	0.0172
5	`read_file`	1,100	4,800	80	0.0045	0.0218
6	`read_file`	1,100	5,900	80	0.0047	0.0265
7	`read_file`	1,000	7,000	80	0.0047	0.0312
8	`read_file` (large file)	1,000	8,000	80	0.0049	0.0361
9	`read_file`	3,500	9,000	80	0.0114	0.0474
10	`read_file`	1,000	12,500	80	0.0058	0.0532
11	`read_file`	1,100	13,500	80	0.0063	0.0595
12	`read_file`	1,000	14,600	80	0.0062	0.0657
13	`run_tests`	1,000	15,600	70	0.0063	0.0720
14	`post_comment`	1,200	16,600	950	0.0158	0.0878
Total		17,800	115,700	2,020		$0.09

Illustrative run of the same 12-file PR, same placeholder base and output prices as above, plus the $2.50/M cache-write and $0.20/M cache-read placeholders above. Not measured on a real agent.

The total drops from $0.29 to about $0.09, roughly a third of the no-cache run. Almost all of that saving comes from one place: 115,700 of the run's 133,500 input tokens show up as cache-read, at a tenth of the base rate, because by the third or fourth step nearly everything the model needs has already been sent once before. Only 17,800 tokens, the new content each step actually adds, ever pay the write premium, and none pay the plain base rate at all: once the breakpoint is marking the whole prefix on every call, there's nowhere for a fresh, uncached token to hide.

Step 9, the large file, still shows up as a cost spike: it adds 3,500 new tokens to write instead of the usual 1,000, so its step cost more than doubles even with caching on. This run's saving comes from what's already cached, not from making the new content cheaper.

This isn't a general "caching cuts your bill by two thirds" result. It holds for this run's hit pattern: a loop that never restarts the cache and never pauses long enough to let it expire. Change either one and the split changes with it, which is exactly what the next section does.

When caching costs you more

The write premium is what makes caching pay off over many steps: pay it once per token, then read that token back at a steep discount on every later step that reuses it. Flip that around, and the same premium is what makes a cache that never hits cost more than not caching at all.

Say the gap between two calls in a run crosses the cache's lifetime, however you get there: a slow tool call, a human approval step between reading the files and posting the comment, a retry after a timeout. The next call's marked prefix no longer matches anything cached. It's a full cache miss on the entire prefix, not just the new part, so the whole thing, not just this step's delta, gets billed as a cache-write, at the premium, on every single step.

Run the same 14-step PR review that way, cache always cold: every call's full input, not just its delta, bills at $2.50 per million instead of $2.00. The run totals about $0.35, more than the $0.29 this same run cost with no caching at all. Turning caching on made this run worse, because the breakpoint kept getting marked but the cache was never actually read, so every token paid the write premium with none of the discount that's supposed to offset it.

The diagnostic is in the log you already have. Sum cache-read tokens and cache-write tokens across the run's rows. A healthy run's cache-read total should dwarf its cache-write total by the last few steps, the way this run's does: 115,700 read against 17,800 written. If cache-read stays at or near zero step after step, while cache-write keeps re-billing the whole growing prefix, the cache isn't holding. Check what's sitting between two calls: a gap longer than the vendor's cache lifetime is the usual cause, and moving the slow step, or extending the lifetime if your plan allows it, is the fix, not turning caching off.

Takeaway

Caching changes what you're billed for the same tokens, not what your loop sends: the PR review agent still sends its whole growing history on every step.
A cached response splits input into three counts, fresh, cache-write and cache-read, each at its own price; pull all three off the usage object before you trust a total.
One cache breakpoint, placed at the end of the growing message list and moved forward on every call, is what turns "everything so far" into a cache hit on the next step.
For this run, caching cut the total from $0.29 to about $0.09; almost all of the saving came from the 115,700 tokens billed as cache-read, not from the 17,800 that still paid the write premium.
A cache that never hits still pays the write premium on every call, and can cost more than no caching at all: about $0.35 here, against $0.29 with caching off entirely.
Check the diagnostic before you trust the discount: cache-read tokens should dwarf cache-write tokens by the run's later steps; if they don't, something sitting between your calls is breaking the cache.