Prompt Caching: Paying Once for the Prefix that Every Request Repeats

The Bill That Grew Faster Than the App

UrbanTreelogy, the gardening app I build solo, has an assistant feature: a chat that answers plant-care questions. Every API call it makes carries the same freight - a long block of instructions about tone and safety, a set of care guidelines, tool definitions for looking up the plant database. Only the last few dozen tokens, the user’s actual question, change from request to request.

When I finally sat down with the usage logs, the shape of the spend was almost embarrassing: the overwhelming majority of input tokens were identical across requests, token for token, and I was paying full price for them every single time. Turning on prompt caching cut the input side of the bill dramatically, with no change to the model, the prompt, or the answers.

This post is about why that works. Prompt caching is not a discount coupon; it is a structural property of transformer inference, surfaced as a billing feature. We will build it up from the KV cache, derive the exact-prefix rule it lives by, work out the economics symbolically, and finish with a small calculator where we can plug in a provider’s current prices and see the expected amortized cost. Deliberately, this post pins down no specific prices, multipliers, or expiry windows beyond one date-stamped worked example: those numbers differ across providers and change often. The mechanism does not.

---

Where the Money Goes

An LLM API bills two meters: input tokens (everything we send) and output tokens (everything the model generates), with output priced several times higher per token. It is tempting to conclude that output is where the money goes. For assistant and agent-style workloads, it often is not.

Look at the anatomy of a typical request:

Segment	Changes between requests?	Typical size
System instructions	No	Hundreds to thousands of tokens
Tool and schema definitions	No	Hundreds to thousands of tokens
Reference content, few-shot examples	Rarely	Hundreds to thousands of tokens
Conversation history	Grows, but past turns do not change	Grows per turn
The new user message	Every request	Tens of tokens

Here is that table in practice: UrbanTreelogy’s plant-diagnosis request, abridged, with rounded token counts per segment:

[system]      You are the plant-care assistant inside a gardening app.
              Reason from symptoms to likely causes, prefer gentle
              remedies first, never recommend restricted pesticides, ...
              (~1,800 tokens of instructions and safety rules)

[tools]       search_plant_db(species, region) -> care profile
              get_user_garden() -> the user's saved plants
              (~700 tokens of JSON schemas)

[guidelines]  Symptom-to-cause reference: yellowing, browning tips,
              leaf drop, pests, root rot, over- and under-watering, ...
              (~2,400 tokens of reference content)

[user]        "My money plant's leaves are turning yellow at the edges.
              I water it every day. The balcony gets full sun."
              (~35 tokens - the only part that changes)

Roughly 4,900 of the request’s 4,935 input tokens, over 99%, are identical from one diagnosis to the next. Only the complaint at the end is new. This is exactly the “identical token for token” shape the usage logs showed.

The response might be a few hundred tokens. The request can carry ten to a hundred times that, and in workloads shaped like the table above, most of it is a fixed preamble we re-send on every call. Without caching, prompt tokens behave like a subscription fee charged per request: the model re-reads, and we re-pay, the same instructions thousands of times a day.

That repetition is not just a billing nuisance. It is also redundant computation, and redundant computation is something we already know how to eliminate.

---

From KV Cache to Prompt Cache

In the KV cache lesson we saw how a model avoids recomputing the past within a single generation: the Key and Value projections of already-processed tokens are stored once and reused at every subsequent step. We were also careful to note its limit - the KV cache is transient, per-request state. It dies with the generation, and an independent request starts from zero.

Prompt caching is what happens when a provider refuses to accept that limit. The closing section of that lesson listed prefix sharing among production cache strategies: requests with a common prefix can share that portion of the cache. Prompt caching is prefix sharing turned into a product - the provider persists the KV state of a prompt prefix for a while and lets later requests load it instead of recomputing it.

Why is this legal, mathematically? Because of causal attention. In a decoder-only transformer, the hidden state at position $i$ - and therefore the K/V projections at position $i$ - depends only on tokens $1$ through $i$. Nothing later feeds back into it. So two requests that share an identical prefix produce identical K/V tensors for that prefix, layer by layer, position by position. The provider can compute them once during the first request’s prefill, store them keyed by the exact prefix, and splice them into the next request that starts the same way. The expensive part of prefill for those positions simply does not run again.

The same argument, read in reverse, explains the sharp edge of the feature: change the token at position $j$ and the hidden states from position $j$ onward all change, so every cached position after the first difference is worthless. There is no fuzzy matching, no “90% similar” credit. Caching is prefix-exact because attention is causal.

Two practical consequences fall out immediately:

• A cache hit is cheaper for the provider, which is why it is cheaper for us: the discount on cached tokens is the provider passing through saved prefill compute.
• A cache hit is faster: skipping prefill for thousands of tokens cuts time-to-first-token, sometimes more noticeably than it cuts the bill.

---

The Prefix Rule: Identical Tokens or Nothing

Everything above compresses into one operational rule, and it is worth stating harshly because most caching failures in practice are violations of it:

Within one model and cache scope, the cache reuses the longest identical token prefix of the assembled model input. From the first token that differs, everything after it is recomputed at full price.

“Assembled model input” matters, twice over. First, the match is on tokens, not raw bytes - though since identical bytes tokenize identically, the practical way to guarantee an identical token prefix is to keep the rendered request byte-for-byte stable. Second, the API assembles our request into one token sequence in a provider-defined order; on several APIs tool definitions render before the system instructions, which is why a tiny edit in a tool schema can invalidate a cached system prompt that we never touched. Caches are also scoped - typically to the model and the account or cache namespace, sometimes to a region - so an identical prefix sent to a different model shares nothing.

The design discipline that follows is stable prefix, volatile suffix:

• Put the content that never changes - instructions, schemas, reference material, few-shot examples - at the front, and keep it frozen.
• Push everything request-specific - the user’s question, retrieved documents, per-request context - to the back.
• Let conversation history grow append-only. Appending preserves the largest possible reusable prefix: earlier turns are read from cache while the newly appended content is processed fresh, and can then extend the cached prefix for the following turn. Rewriting history (say, summarizing old turns in place) rewrites the prefix and forfeits it - sometimes a worthwhile trade, since long conversations eventually need compaction to control context length, but a trade to make consciously.

Providers differ in the plumbing: some cache automatically whenever they detect a repeated prefix, others ask us to mark explicit cache breakpoints in the request, usually with a small limit on how many we may place. Two more provider-specific parameters matter and are worth looking up rather than assuming:

• A minimum cacheable length. Prefixes below some token count are typically processed uncached (an explicit cache request may instead be rejected, depending on the API), and the miss is usually silent - the request succeeds, the meter just never shows a hit.
• A time-to-live (TTL). Cached prefixes expire after a window of minutes to hours; on some platforms a hit refreshes the clock, on others the window is fixed. Traffic sparser than the TTL never reuses anything.

---

The Economics of a Cache Hit

Caching is not automatically free money. Most APIs discount cache reads; on the write side they differ. Some charge a premium to write a prefix into the cache, others fill an automatic cache at the ordinary input rate, and explicit cache objects on some platforms add time-based storage charges on top. The derivation below covers the common read/write-multiplier model: set $w = 1$ for an API with no write premium; storage-billed caches add a time term that we fold into $w$ in the worked example below. Let us keep everything symbolic:

Symbol	Meaning
$p_\text{in}$, $p_\text{out}$	Base price per input and output token
$w$	Cache write multiplier, $w \geq 1$
$r$	Cache read multiplier, $r \ll 1$
$C$	Cacheable prefix length, in tokens
$F$	Fresh (volatile) input tokens per request
$G$	Output tokens per request
$h$	Cache hit rate: fraction of requests finding the prefix cached
$n$	Number of requests reusing one written prefix within its TTL

Reuse-count view. Suppose a prefix of $C$ tokens is written once and read $n-1$ times before it expires. Compared with $n$ uncached requests:

\[\underbrace{C\,p_\text{in}\big(w + (n-1)\,r\big)}_{\text{with caching}} \;<\; \underbrace{n\,C\,p_\text{in}}_{\text{without}} \quad\Longleftrightarrow\quad n \;>\; \frac{w - r}{1 - r}.\]

With $w$ at or modestly above one and $r$ far below one, the right-hand side lands barely above one for typical pricing: a single reuse within the TTL is often already profitable, and every reuse after that is nearly free relative to base price. This is the punchline that makes prompt caching unusual among optimizations - the break-even point is almost immediately behind us.

A concrete instance. As of August 2026, the Gemini API pricing page lists Gemini 3.6 Flash at \$0.75 per million input tokens, \$3.75 per million output tokens (thinking included), and context-cache reads at \$0.075 per million, so $r = 0.1$. No write premium is listed, so $w = 1$ - but cache storage bills at \$0.50 per million tokens per hour, exactly the term our multiplier model lacks. The fix is one substitution: storage at rate $s$ per token-hour, held for $T$ hours, simply acts like a bigger write multiplier. For a one-hour hold:

\[w_{\text{eff}} \;=\; 1 + \frac{s}{p_\text{in}}\,T \;=\; 1 + \frac{0.50}{0.75} \times 1 \;\approx\; 1.67, \qquad \frac{w_{\text{eff}} - r}{1 - r} \;=\; \frac{1.67 - 0.1}{0.9} \;\approx\; 1.74\]

Break-even sits at 1.74: the cache pays for itself, storage included, from the second request inside the hour. Scale that to the 4,900-token diagnosis prefix from earlier, with twenty requests arriving in one hour. To keep the arithmetic readable, count costs in units of “the prefix sent once, uncached” (about \$0.0037 for this prefix):

• Without caching: all twenty requests pay full price for the prefix: $20$ units.
• With caching: request 1 creates the cache, billed at the ordinary input rate: $1$ unit.
• Storing the prefix for the hour: $0.50 / 0.75 \approx 0.67$ units - the storage part of $w_{\text{eff}}$.
• Requests 2 through 20 are nineteen cache reads at $r = 0.1$ each: $1.9$ units.
• Total with caching: $1 + 0.67 + 1.9 = 3.57$ units instead of $20$ - an 82% cut on the cacheable tokens. (The fresh user tokens and the output bill the same either way, so the total-bill cut is smaller; the calculator below shows both numbers.)

The same page also shows why this post keeps its formulas symbolic: it already announces that every one of those prices doubles on January 1, 2027. The arithmetic survives; the inputs do not.

Hit-rate view. Steady-state traffic is easier to reason about per request. If a fraction $h$ of requests hit the cache (paying $r$ per cached token) and the rest miss and re-write it (paying $w$), the expected multiplier on each cacheable token is

\[m \;=\; (1-h)\,w + h\,r,\]

and caching beats no caching exactly when $m < 1$. Solving that for $h$ gives the break-even hit rate $h^{\ast}$, the threshold above which read discounts outweigh write premiums:

\[h \;>\; h^{\ast} \;=\; \frac{w - 1}{w - r}.\]

If a provider charges no write premium at all ($w = 1$), then $h^{\ast} = 0$ and any nonzero hit rate helps. With a write premium, $h^{\ast}$ sits somewhere in the low tens of percent for typical multipliers - a bar that a busy chat assistant with append-only history tends to clear easily, and that a sparse batch job running once a day may never clear at all.

Putting it together, the expected cost of one request is

\[\mathbb{E}[\text{cost}] \;=\; p_\text{in}\,\big(F + C\,m\big) \;+\; p_\text{out}\,G,\]

with the output term untouched: caching does nothing for output tokens. That is the formula the calculator below implements.

---

Plug In Real Numbers: an Amortized-Cost Calculator

The symbols become concrete the moment we substitute one provider’s actual price sheet. Enter the base prices per million tokens (any currency - results come out in the same units), the provider’s write and read multipliers ($w = 1$ for an API with no write premium), the shape of a typical request, and the hit rate we expect within the TTL. The optional minimum-length field marks the provider’s eligibility threshold: a cacheable prefix below it is not cached at all. The defaults are illustrative placeholders, not a recommendation; overwrite all of them. One scope note: the calculator models token-priced cache writes and reads only - explicit cache objects that bill storage per unit time need that extra term added on top, or folded into the write multiplier as $w = 1 + (s/p_\text{in})\,T$ the way the worked example above does.

Amortized cost calculator

Currency-agnostic: results are in the same units as the prices entered. Token-priced cache writes and reads only; time-based storage charges are not modeled.

Input price / 1M tokens

Output price / 1M tokens

Cache write multiplier w

Cache read multiplier r

Input tokens / request

Output tokens / request

Cacheable prefix, % of input

Cache hit rate within TTL, %

Min cacheable prefix, tokens (optional)

Uncached, per 1,000 requests

-

With caching, per 1,000 requests

-

Savings, total

-

Savings, input side

-

-

Two experiments worth running in it: drag the hit rate down and watch the savings evaporate as misses (each paying the write premium) dominate; and drag the cacheable share down to see why a prompt with its stable content scattered through the middle, rather than consolidated at the front, caches poorly no matter how good the hit rate is.

---

What Silently Kills the Hit Rate

Prompt caching fails quietly. The request still succeeds, the answer is still fine, and the meter simply bills full price. Most failure modes are self-inflicted violations of the identical-prefix rule, usually invisible ones; the last two below are lifecycle misses that can hit even a perfectly stable prefix:

• A timestamp in the system prompt. Interpolating the current date or time into the instructions makes every request’s prefix unique. If the model needs the date, append it near the end of the request instead.
• Random identifiers. Request IDs, session IDs, trace tokens - anything generated fresh per request that lands in the prefix.
• Non-deterministic serialization. Building tool schemas or config blobs from a hash map whose key order varies run to run produces semantically identical JSON whose bytes, and therefore tokens, differ. Sort the keys.
• Per-user content placed early. A personalization block at the top of the system prompt means no two users ever share a prefix. Shared instructions first, user-specific material last.
• Rotating prompt variants. A/B testing two phrasings of the instructions, or deploying prompt tweaks several times a day, splits and invalidates the cache each time.
• History rewriting. Summarizing or pruning old conversation turns in place; the suffix we saved is a prefix we broke.
• Traffic sparser than the TTL. If requests arrive further apart than the cache lifetime, every request is a fresh write. With a write premium that actively costs more than not caching; with free writes it simply saves nothing.
• Provider-side lifecycle. Even an identical prefix can miss: entries expire at the TTL, can be evicted under capacity pressure, and are scoped per model and namespace - so a model upgrade, a region change, or a routing quirk starts the cache cold.

The defense is not intuition; it is the meter. APIs that offer caching expose usage fields on each response: a cached or read-token count is common, while a separate cache-write count exists on some platforms but is not universal. Log whatever the provider reports. A healthy integration shows cache reads covering most input tokens after the first request. a broken one shows no reads at all (or a write on every call, where writes are reported). This is exactly how I caught UrbanTreelogy’s own mistake - a version string interpolated into the instructions was quietly zeroing the hit rate, and the usage fields, not the bill, revealed it within an hour.

---

What Prompt Caching Does Not Change

A cache hit changes where the prefix’s KV state comes from, and nothing else. Four boundaries worth keeping straight:

• The context window is unchanged. Cached tokens still occupy their positions in the context; caching is not a way to fit more in.
• The model still conditions on every cached token. Answers are byte-for-byte as they would have been without caching: quality neither improves nor degrades, and cached instructions steer the model exactly as before.
• Rate-limit accounting is provider-specific. Some platforms count cached tokens against throughput limits at full weight, others discount them; the pricing answer and the rate-limit answer can differ.
• A cache is retained state. The prefix persists on the provider’s side for its lifetime, and longer retention modes can interact with data-retention or residency commitments - a governance question as much as a billing one.

---

A Deployment Checklist

Everything in this post folds into a few lines to check before and after enabling caching:

1. Put the fixed content first, the changing content last: tools and schemas, then instructions and reference content, then conversation history, then the new user message.
2. Keep the fixed part truly fixed. Do not interpolate the current date, timestamps, request IDs, or anything random into it; serialize tool schemas and config the same way every time (sort the JSON keys); move per-user personalization to the end of the request, after the shared content.
3. Grow history append-only; treat any history rewrite as a deliberate cache purchase.
4. Check the provider’s parameters against current docs, not memory or a blog post (this one included): the caching mode (automatic, explicit breakpoints, or cache objects), read, write, and any storage pricing, the TTL and whether hits refresh it, the minimum cacheable length, cache scoping across models and regions, how cached tokens count against rate limits, and any data-retention implications.
5. Check that caching is actually happening. Each API response comes with usage numbers that say how many input tokens were served from cache; once things are warmed up, that should be most of them. Do not wait for the monthly bill to find out.
6. Re-run the arithmetic when pricing changes; the calculator above is the two-minute version.

Stepping back, prompt caching is the KV cache lesson applied one level up. Within a single generation, the model reuses the K/V state of tokens it has already processed instead of recomputing them. Prompt caching keeps that state around so the next request can reuse it too. Same mechanism at two scales: inside one request it saves time, across requests it saves money.

With the cost of sending context under control, the natural next question is whether the context we send - especially the retrieved documents we stuff into it - is actually any good. That is the subject of the next lesson, RAG evaluation.

---

Resources

The concept in this post is stable; the parameters are not. The authoritative pages for three widely used APIs, each documenting its own activation mode, multipliers, TTL, minimum lengths, and usage fields, plus the pricing page behind the worked example:

• Prompt caching - Anthropic docs
• Prompt caching - OpenAI docs
• Context caching - Gemini API docs
• Gemini API pricing - source of the worked example’s numbers, including their scheduled 2027 change