You ran the meter from Put a meter on your agent on an agent of your own, and watched input tokens climb every single step. Here it's a customer-support agent working one escalating refund ticket. It has three tools, lookup_order, check_refund_policy and search_past_tickets, and it takes 14 steps of back-and-forth with the customer to close the ticket. Step 1 sends about 900 input tokens. Step 14 sends about 8,250, roughly nine times as much.
The whole run comes to about 69,000 input tokens and $0.16. Prompt caching would discount what that costs to resend. It would not shrink what gets resent, and this post is about the thing caching leaves alone.
Three ways to send less in the first place: drop the old tool results the agent probably doesn't need in full, compress the older steps into a running summary, or fetch old context back only on the step that actually asks for it. Same run, four totals, run by the same deterministic estimator throughout so the comparison is reproducible without an API key.
Trimming: keep the call, drop the body
The cheapest fix is a sliding window. Keep the last four steps' tool results in full; once a tool result is older than that, replace its body with a one-line marker. The agent still sees that it called lookup_order at step 2, it just doesn't see that call's full 800-token output anymore, past step 6.
def trim_history(history, keep_full=4):
"""history: a list of steps, each {"step", "role", "tool", "text"}.
Tool results older than the last `keep_full` steps get replaced by
a one-line marker; every other message, and the fact the call
happened, is kept as-is."""
latest = history[-1]["step"]
trimmed = []
for msg in history:
is_old_result = msg["tool"] and (latest - msg["step"] >= keep_full)
if is_old_result:
marker = f'[{msg["tool"]} result, step {msg["step"]}, dropped]'
trimmed.append({**msg, "text": marker})
else:
trimmed.append(msg)
return trimmed
Run trim_history on the ticket's full 14-step trace with keep_full=4, and print which tool results survive. Five steps called a tool: 2, 3, 6, 9 and 12. Measured from step 14, everything before step 10 is older than the window:
step 2 lookup_order -> [lookup_order result, step 2, dropped]
step 3 check_refund_policy -> [check_refund_policy result, step 3, dropped]
step 6 search_past_tickets -> [search_past_tickets result, step 6, dropped]
step 9 lookup_order -> [lookup_order result, step 9, dropped]
step 12 check_refund_policy -> kept in full (within the last 4 steps)
Every customer message and every assistant reply stays untouched. trim_history only ever touches text on a tool-result message; it never removes the message, so the record that the agent made a call, and when, survives even after the body is gone.
That's also where trimming breaks. Say the customer's fifth message references the order details from that first lookup_order call, already dropped by then. The agent's context now holds [lookup_order result, step 2, dropped] where the item, size and payment method used to be. It can't answer from memory anymore. It has to call lookup_order again, or answer wrong. Call this the recall tax: what a trimmed detail costs you to get back, the moment you need it and it's gone.
A single window size doesn't fit every tool. search_past_tickets output rarely matters again once the agent has read it; four steps is generous. lookup_order output gets asked about again constantly in a refund flow, order number, item, payment method, all come back mid-conversation. A tool-by-tool policy, not one rule for every tool, is what keeps the recall tax from landing on the tool that can least afford it.
Summarising: compress instead of drop
Trimming throws the body away. Summarising keeps a compressed version of it. Every four steps, the oldest un-summarised block collapses into a running summary, regenerated from the prior summary plus the newly-collapsed steps, through one extra model call.
def summarize_history(prior_summary, block, call_summarizer):
"""Collapse `block` (the oldest un-summarised steps) into a new
running summary. `call_summarizer` is your model call, stubbed
here; its own input and output tokens count toward the run total,
same as any other step."""
block_text = "\n".join(f'step {m["step"]}: {m["text"]}' for m in block)
prompt = f"Prior summary:\n{prior_summary}\n\nNew steps:\n{block_text}"
new_summary, tokens_in, tokens_out = call_summarizer(prompt)
return new_summary, tokens_in, tokens_out
On this run, call_summarizer fires three times: after step 4, after step 8, and after step 12. Each call's own tokens go into the run's running total, the same as a lookup_order call's would:
after step 4: prior summary 0 + block 2,450 -> summary ~180 tokens out
after step 8: prior summary 180 + block 2,550 -> summary ~220 tokens out
after step 12: prior summary 220 + block 1,950 -> summary ~260 tokens out
By step 14, the agent's context holds a roughly 260-token summary standing in for steps 1 through 12, plus steps 13 and 14 in full since they haven't hit a summarisation boundary yet. That's a lot smaller than resending 12 raw steps. It's also not free: those three extra calls add about 7,350 input tokens and 660 output tokens to the run, on top of whatever the main loop itself sends. Skip that accounting and summarising looks cheaper than it is.
What the summary buys back that a plain marker doesn't is the narrative. The summary can say "customer requested a refund for order #48213, approved under the late-delivery policy, prior similar ticket on file recommending a full refund over store credit," which a one-line marker per dropped tool call can't. What it can't do is answer a question about an exact figure it compressed away. If the customer later asks for the precise delivery date from the original lookup_order call, the summary might not have kept that detail; the recall tax from the trimming section applies here too, just paid in lost precision instead of a dropped message.
Retrieval: fetch it back only when asked
Trimming and summarising both make a decision up front, at write time, about what the agent won't see again. Retrieval defers that decision to read time: drop tool results to a marker by default, same as trimming, but give the agent a tool to pull the original body back on whichever step actually needs it.
def search_history(history, query, top_k=2):
"""Keyword match: score each stored tool result by the words it
shares with the query, return the top_k. This misses paraphrases
on purpose, it only matches words that literally appear in both."""
terms = set(query.lower().split())
scored = []
for msg in history:
if not msg["tool"]:
continue
overlap = len(terms & set(msg["text"].lower().split()))
if overlap:
scored.append((overlap, msg))
scored.sort(key=lambda pair: -pair[0])
return [msg for _, msg in scored[:top_k]]
On this run, only one step needs an old detail back: at step 11 the customer asks how long the refund will take, and the answer, the 5-to-7-business-day window, lives in check_refund_policy's output from step 3, long since dropped to a marker. The agent calls search_history(history, "refund processing time"), gets that one stored result back, about 150 tokens, uses it to answer, and the fetched text is not added back into the growing history. It's spent once, on the step that asked, and gone again.
That's the whole difference from trimming: trimming's window means an old tool result is either fully present or fully gone, for every future step. Retrieval means it's gone by default and reappears, briefly, exactly when something asks the right question. Over this run, that default-off stance is what saves the most.
search_history here is a keyword match over stored tool results, nothing more: no embeddings, no vector database, no extra service to run. Treat it as a stand-in for what production retrieval usually adds: an embedding model to turn text into vectors, and a vector store to search them by similarity instead of shared words. The gap shows up exactly where you'd expect. Ask search_history about "refund processing time" and it finds the passage that contains those words. Ask it about "how fast will I get my money back," and it may not, because nothing in the stored text matches those words literally. Embedding-based retrieval is built to catch that; keyword matching is not.
Same run, four totals
Apply all three techniques to the same 14-step trace, run estimate_tokens over each one's resulting history, and price the totals at the same placeholder rates as the meter post: $2 per million input tokens, $10 per million output. These are the prices behind the $0.16 baseline in the opening.
Strategy | Input tokens | Output tokens | Cost | Cost savings vs. baseline |
|---|---|---|---|---|
Baseline (nothing shrinks) | ~69,300 | ~1,900 | ~$0.16 | n/a |
Trim (window = 4 steps) | ~46,900 | ~1,900 | ~$0.11 | ~31% |
Summarise (every 4 steps) | ~36,700 | ~2,560 | ~$0.10 | ~38% |
Retrieve (fetch on demand) | ~33,900 | ~1,940 | ~$0.09 | ~44% |
All four numbers come from a fixed characters-per-token estimator, not a vendor tokenizer, so treat them as directional, not billing-accurate; a real tokenizer's count on the same text will differ by some margin, and the exact figure to trust is whatever your provider's usage object reports. What won't move much is the shape: retrieve beats summarise beats trim beats baseline, because this run's shape rewards not resending tool output by default, more than it rewards keeping a lossy trace of it around.
That ordering is specific to this run's shape, a mix of a few sizeable tool results and a lot of back-and-forth conversational turns. If most of your history is tool output, trimming and retrieval win by more: there is more body to cut. An agent that's almost entirely conversation, with few or no tool calls, would get little from any of the three; there's not much tool-result weight to drop in the first place. Meter your own agent's shape before assuming this run's order carries over.
Which one, when
None of the three is a strict upgrade over the others; each pays the recall tax differently.
Trim when old tool output is rarely needed again, and when the tool's exact wording rarely matters once the agent has acted on it. Cheapest to build, and the failure mode is loud: a dropped-marker message that has to be re-fetched, not a silently wrong answer.
Summarise when the narrative across many steps matters more than any single old detail: a long-running ticket where "what happened so far" needs to survive, even if some numbers along the way don't.
Retrieve when specific old details come up again, but only occasionally and unpredictably, so it's cheaper to fetch them on the rare step that asks than to keep them resent on every step that doesn't.
They also combine. Trim aggressively by default, since most old tool output never gets asked about again, and let search_history fetch back exactly what a later step turns out to need. That's closer to what this run's numbers already show: retrieval's win here came from treating "dropped by default, fetched on demand" as the starting stance, not an afterthought bolted onto trimming.
One thing all three leave alone: none of this changes what a call is billed for once it's sent, only how much gets sent.
Takeaway
Trimming, summarising and retrieval all cut what an agent resends every step, but none of them is free: trimming costs recall on the tool result it drops, summarising costs precision plus the tokens of the call that keeps it current, retrieval costs a paraphrase it can't match.
On this run, cutting what gets sent saved more than caching what gets sent: retrieve, the cheapest of the three, brought the run from about $0.16 to about $0.09, while caching alone only discounts the $0.16.
The three combine. Trim aggressively, and let a small
search_historytool fetch back exactly what a later step asks for.Every number here comes from a deterministic estimator, not a vendor tokenizer, and from reasoning through that estimator rather than running it in this session. Run it on a trace of your own agent before you trust the order these four totals came out in.
