Every time you spin off your agent, it has to re-read its whole life story before performing an action. Here’s how to make it remember without having to feed it the same context again.
In the movie Groundhog Day, Bill Murray’s character re-lives the same February 2 every day. He wakes up to the same song playing on the radio, in the same inn, and the same small town, again and again for what some movie fans estimate to be 34 years.
Nothing he accomplished the day before carries over unless he works it out again from scratch. AI agents live in a version of Groundhog Day. Every time an agent takes a step (calls a tool, reads the result, decides what to do next), it has to send the model its entire story so far to maintain continuity: the tool definitions, the instructions, every previous tool call, and every result. The model only predicts the next token based on the input you provide. It has no internal memory between requests, so it must read all of it again. By turn 20 of a coding session, the model re-reads turns 1–19, and it’s you who pays for every one of those tokens, again and again.
Prompt caching breaks the cycle. The service stores your input prompt (instructions, tools, and content) and reuses it on the next request, at a fraction of the input price.
In this post, I walk you through an agent’s life, from its first turn to when it gets new tools and spawns helpers. Along the way, I show you how prompt caching keeps costs down and keeps agents responsive and consistent.
What an agent sends to its LLM
Before talking about caching, it helps to look at what a typical agent’s first invocation to the large language model (LLM) contains:
- Tool list – The definitions of every function the agent can call. Changes when you ship a new release.
- Instructions – The developer prompt: rules, persona, policies, examples. Also changes per release.
- History – Everything that happened in this session so far: the task, the model’s tool calls, and the tool results (which grows every turn).
- New turn – The latest tool result or user message (which might be different every time).
What an agent sends to the model, turn after turn if you don’t set up caching:
Figure 1: What an agent sends to the model on every turn
How prompt caching works under the hood
When a model reads a prompt, it first does a write called prefill. During prefill, it processes every input token and builds its key and value vectors (also known as key-value, or KV), which form the model’s working state. This happens before the model writes its first token.
Prompt caching saves that state, which contains the system prompt, tools, content, history, and so on. When a later request starts with the same input tokens (that is, the same text), the prompt caching service loads the saved state instead of computing it again. It then processes only what is new in the next payload to the model. Prompt caching can be implicit, where Amazon Bedrock selects where the reusable prefix ends, or explicit, where your code marks that endpoint using prompt_cache_breakpoint. You can use both approaches together, and we recommend doing so.
Turn 1 compared to turn 2
Figure 2: Turn 1 saves the prompt prefix up to the breakpoint, and turn 2 reuses it and pays full price only for the new question
Implicit or explicit, and why do we need both?
For agentic workloads, implicit caching lets Bedrock choose what to cache as the conversation grows. With explicit caching, you can mark shared content, such as tools and instructions, for reuse across sessions and when spinning up helper agents. You don’t always need both forms of caching, but combining them gives you a reusable starting point while Bedrock manages the growing conversation automatically.
New sessions can also reuse that shared starting point while each conversation builds on its own cached history. In this post, we start with the first turn, then show how to carry cached content through later turns, new tools, and across subagents.
Hello cached world
The first win is to cache the static part of the prompt: the tools and instructions that stay the same across requests. These examples use your existing Amazon Bedrock client, TOOLS, and task. Replace the shortened INSTRUCTIONS string with your real policies and examples. The reusable prefix must reach 1,024 tokens to qualify.
When caching static content, use explicit mode: choose where the saved prefix ends.
KEY = "support-agent:v7:gpt-6-astra"
INSTRUCTIONS = "Your full support policies and examples..."
# Rendered context: [provider context][TOOLS][INSTRUCTIONS] | [task]
# Cache starts at the beginning; the marker below chooses its end.
resp = client.responses.create(
model="us.openai.gpt-6-astra",
store=False, # Response storage; prompt caching still works.
include=["reasoning.encrypted_content"],
prompt_cache_key=KEY,
prompt_cache_options={"mode": "explicit", "ttl": "30m"},
tools=TOOLS, # Rendered before instructions; keep order stable.
input=[
{
"role": "developer",
"content": [{
"type": "input_text",
"text": INSTRUCTIONS,
# Save through this whole block, including TOOLS.
"prompt_cache_breakpoint": {"mode": "explicit"},
}],
},
# User message is processed but not cached.
{"role": "user", "content": task},
],
)The prompt_cache_key tells the service which requests are meant to share a cache. We come back to how to use it when we get to reading the cache in new sessions. In one of our tests using explicit caching, the roughly 4,000-token tools-and-instructions prefix wrote 4,120 tokens on the first request and read those 4,120 tokens on later requests. The task (or user request) comes after the cache boundary, so this doesn’t trigger a write to the cache.
Keep in mind that prompt caching works only when you pass exactly the same text. Make sure that the agent’s instructions and tool lists are serialized the same way every time. Model Context Protocol (MCP) servers and other forms of tool delivery need to be indexed to make sure they stay consistent. Keep anything that varies, such as the date, the user’s name, or a request ID, after the cached instructions.
As the content grows and varies (different tools and content), use implicit mode to let the service follow the conversation. Here, the first request has no explicit marker. Its latest eligible message is the user’s task, so the automatic breakpoint comes after that task and covers the tools, instructions, and task together. The mode is shown explicitly in the following example for clarity, even though implicit is the default. Setting store=False leaves conversation history in your application. It doesn’t disable caching.
def ask_with_implicit_cache(history):
return client.responses.create(
model="us.openai.gpt-6-astra",
store=False,
include=["reasoning.encrypted_content"],
prompt_cache_key=KEY,
prompt_cache_options={"mode": "implicit", "ttl": "30m"},
tools=TOOLS,
input=history, # Everything gets cached
)
history = [
{"role": "developer", "content": INSTRUCTIONS},
{"role": "user", "content": task},
]
# Automatic endpoint: the end of the user message.
resp = ask_with_implicit_cache(history)This is useful when the next turn continues the same task. A different task doesn’t match the prefix through the old task, and the implicit write doesn’t necessarily create a separate entry for only the tools and instructions. Use an explicit marker there when you need that shorter shared prefix to be reusable across tasks.
Use both when the agent needs a shared starting point and a growing history. Start with the marked developer block in the following code example and call the function ask_with_implicit_cache. The request-level mode stays implicit. The block-level marker adds an explicit save point after the instructions. The service can also save through the latest eligible message. This keeps the shared prefix available while the conversation grows.
history = [
{
"role": "developer",
"content": [{
"type": "input_text",
"text": INSTRUCTIONS,
"prompt_cache_breakpoint": {"mode": "explicit"},
}],
},
{"role": "user", "content": task},
]
# Still implicit mode, now with an explicit startup save point in the history.
resp = ask_with_implicit_cache(history)Caching conversation history and tool results
The model requests tools by name and supplies their arguments. Your application must execute them. execute_function is a helper you build to route those requests to the matching Python functions and return their results. The following loop uses this helper to run each requested tool. It appends the model’s output and all tool results to the history list, then sends that list again until the model returns a final answer. In implicit mode, Amazon Bedrock can reuse the unchanged prefix and cache the added content through the last tool result. Keep earlier messages and the initial TOOLS unchanged.
import json
# Map tool names to their Python implementations.
TOOL_HANDLERS = {
"get_owner": get_owner,
# Register your other initial tools here.
}
def execute_function(name, arguments):
"""Run the requested tool and return its result."""
handler = TOOL_HANDLERS.get(name)
if handler is None:
return {"error": f"Unknown tool: {name}"}
return handler(**arguments)
while True:
# Preserve all output items, including encrypted reasoning.
history.extend(resp.output)
calls = [x for x in resp.output if x.type == "function_call"]
if not calls:
break
for call in calls:
result = execute_function(call.name, json.loads(call.arguments))
history.append({
"type": "function_call_output",
"call_id": call.call_id,
"output": json.dumps(result),
})
# The last result in this group supplies the automatic endpoint.
resp = ask_with_implicit_cache(history)For follow-ups, append a new user message and call the helper again. Keep earlier messages unchanged so their cached endpoints remain available.
Rolling breakpoints: Each turn reads everything before it
Figure 3: Rolling breakpoints, where each turn reads everything before it and writes only its own tool result
For a separate explicit-only run, recreate the two-message history from the combined Hello cached world example. This loop wraps tool results in input_text blocks and marks the final result in each batch. Earlier markers stay in history. See the current lookup rules for long conversations.
import json
while True:
resp = client.responses.create(
model="us.openai.gpt-6-astra", store=False,
include=["reasoning.encrypted_content"],
tools=TOOLS, prompt_cache_key=KEY, input=history,
prompt_cache_options={"mode": "explicit", "ttl": "30m"},
)
d = resp.usage.input_tokens_details
print("read", d.cached_tokens, "write", d.cache_write_tokens)
history.extend(resp.output) # Includes encrypted reasoning.
calls = [x for x in resp.output if x.type == "function_call"]
if not calls:
break
batch = []
for call in calls:
result = execute_function(call.name, json.loads(call.arguments))
batch.append({
"type": "function_call_output", "call_id": call.call_id,
"output": [{"type": "input_text", "text": json.dumps(result)}],
})
# Add the newest boundary; retain every earlier marker.
batch[-1]["output"][-1]["prompt_cache_breakpoint"] = {"mode": "explicit"}
history.extend(batch)GPT-6 Astra tied for the best implicit result in this six-turn workload: about 62 percent lower input cost than running without caching. Explicit rolling breakpoints saved about 61 percent on the same model. Both reused more history as the conversation grew. These are workload results. Measure your own agent’s savings.
Figure 4: Input cost of every caching setup on GPT-6 Astra, relative to no caching
Watch out for:
- Only a static breakpoint in explicit-only mode – The tools and instructions are cached, but the growing history is processed again on every turn (0.61). This limitation doesn’t apply to the combined example, which also saves through eligible conversation endings.
- Moving a single breakpoint can prevent cache reuse and cause repeated writes in explicit-only mode. Keep the previous turn’s breakpoint too.
- Only an implicit write through a changing task – A new task might share the tools and instructions but not the previously saved endpoint. Add an explicit marker after the shared instructions if different tasks need to reuse that shorter prefix.
When tools change
Sometimes an agent’s tool list changes, or new tools get added. Keep the initial list stable: adding, removing, or reordering definitions can invalidate the cached prefix. Choose the approach that fits how the tool becomes available.
When known tools are invoked in different steps of the agent’s flow: Send the same sorted TOOLS and use allowed_tools to limit what the model can call. With allowed_tools, "mode": "auto" lets the model decide whether to call an allowed tool, and "mode": "required" requires at least one allowed tool call. Pass this as tool_choice. The restriction doesn’t change the cached catalog:
tool_choice={"type": "allowed_tools", "mode": "auto",
"tools": [{"type": "function", "name": "get_owner"}]}New tools during a session. Both Amazon Bedrock Runtime and Amazon Bedrock Mantle endpoints accept an input item type called additional_tools for GPT-6 Astra, GPT-6.1 Sol, and GPT-5.6 Luna. For model availability by AWS Region, see Supported models by AWS Region in Amazon Bedrock. Register the new tool get_due_date in the helper function execute_function, then append its schema to the completed history from the agent loop. Keep the initial TOOLS unchanged:
TOOL_HANDLERS["get_due_date"] = get_due_date
NEW_TOOL = {
"type": "function", "name": "get_due_date",
"description": "Return the due date for a support case.",
"strict": True,
"parameters": {
"type": "object",
"properties": {"case_id": {"type": "string"}},
"required": ["case_id"], "additionalProperties": False,
},
}
history.append({
"type": "additional_tools", "role": "developer",
"tools": [NEW_TOOL],
})
history.append({
"role": "user", "content": "Find the due date for case C-123.",
})
# Implicit mode saves through this new user message, including NEW_TOOL.
resp = ask_with_implicit_cache(history)
d = resp.usage.input_tokens_details
print("old prefix read", d.cached_tokens, "new write", d.cache_write_tokens)
# Example output:
# old prefix read 2048 new write 256A new tool arrives mid-session
Figure 5: Appending a tool keeps the earlier prefix reusable, with example counts from GPT-6 Astra on Amazon Bedrock Runtime
Continue the tool-execution loop with the new response. The loop adds the model’s output to history, runs any requested tools, and appends their results. Keep the additional_tools item in later requests so the new tool remains available. In implicit mode, Amazon Bedrock can cache through the new user message, including the tool definitions that precede it. In exp