What is prompt caching?

Prompt caching reuses the processed form of a prompt's opening tokens when a new request starts with the same text. The model then only processes the new part. Long system prompts, tool definitions and documents that repeat across requests get processed once and reused while the cache is warm.

Do I need to enable prompt caching?

No. Caching is automatic for supported models. You get the most from it by putting stable content at the start of your prompt and variable content at the end. Each response shows how many tokens came from the cache, so you can check that it works.

How do I see whether my request used the cache?

Read `usage.prompt_tokens_details.cached_tokens` in chat completions, `usage.input_tokens_details.cached_tokens` in the Responses API, or `usage.cache_read_input_tokens` in the Messages format. A value above zero means part of the prompt came from the cache; zero means the whole prompt was processed afresh.

Why am I getting no cache hits?

Usually something near the start of the prompt changes on every request, such as the current date, a request id or a user name in the system prompt. Move such details to the end. Also check that the prompt is long enough to qualify and that requests are not too far apart in time.

Does caching affect the quality of answers?

No. The model reads exactly the same tokens with or without the cache. Caching only skips repeating work on the shared opening of the prompt, so answers are the same kind you would get without it.

Is cached data shared with other customers?

No. Cache entries are kept separate per organisation and are only used to serve your own requests. They are held in memory in European data centres, expire after a period without use, and are never used to train models.

Does prompt caching work with the Anthropic format?

Yes. Requests to `/v1/messages` benefit from automatic caching, and clients that add `cache_control` to content blocks, as many Anthropic tools do, work without changes. Cache usage is reported in `cache_read_input_tokens`, as Anthropic clients expect.

Are cached tokens cheaper?

How cached tokens are billed depends on your plan, and we confirm it with you before you go live. Regardless of billing, caching makes long repeated prompts quicker to start answering, which is noticeable in chat applications and agent loops.

Prompt caching for LLM APIs · faster repeated prompts in the EU

cached_prompt.py

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.inference.eu/v1", api_key=os.environ["INFERENCE_EU_API_KEY"])

# Long, stable prefix first: policy text that is the same for every question
POLICY = open("travel-expenses-policy.md").read()

def ask(question: str):
    reply = client.chat.completions.create(
        model="gpt-oss-120b",
        messages=[
            {"role": "system", "content": "Answer questions about the travel policy below. Quote the section number.\n\n" + POLICY},
            {"role": "user", "content": question},  # variable part last
        ],
    )
    cached = reply.usage.prompt_tokens_details.cached_tokens
    print(f"{cached} of {reply.usage.prompt_tokens} prompt tokens came from the cache")
    return reply.choices[0].message.content

ask("Can I book business class for a six-hour flight?")
ask("What is the daily meal allowance in Germany?")

How prompt caching works

Before a model writes anything, it processes the whole prompt. When a new request begins with exactly the same tokens as a recent one, the API can reuse the processed state for that shared prefix and only process what is new. This is called prefix caching: it matches from the first token onwards and stops at the first difference.

Caching is automatic for supported models on the chat completions, Responses and Messages endpoints, with no extra fields needed. The cache lives in memory in the same European data centres that process your requests, is kept separate between organisations, and expires after a period without use.

Structuring prompts for cache hits

Put what stays the same first and what changes last. A good order is: system prompt, tool definitions, long reference documents, earlier conversation turns, and finally the new user message. Even a small change near the start, such as a timestamp or user name in the system prompt, makes everything after it a miss.

Keep the stable part byte-for-byte identical: the same whitespace, the same order of tools, the same JSON serialisation. In chat applications, append new turns rather than rewriting history, so each request extends the cached prefix of the one before. Send related requests close together in time while the cache is still warm.

Measuring the effect

Every response reports how many prompt tokens came from the cache. In chat completions read usage.prompt_tokens_details.cached_tokens, in the Responses API usage.input_tokens_details.cached_tokens, and in the Messages format usage.cache_read_input_tokens. Log this alongside total prompt tokens to see your hit rate per feature.

A low hit rate usually means something at the start of the prompt changes on every call. Compare two consecutive prompts and look for the first difference. Anthropic-format clients that send cache_control markers on content blocks work unchanged; the markers are accepted. How cached tokens are reflected on your invoice depends on your plan, and we confirm it with you.

Limits and edge cases

Very short prompts are not worth caching, so caching only applies above a minimum prompt length that depends on the model. Cache entries are evicted after a period of inactivity and under heavy load, so a hit is likely but never guaranteed; your application must work the same either way.

Caching does not change the output. The model reads the same tokens and produces the same kind of answer; only the work on the shared prefix is skipped. Cached data is used only to serve your organisation's requests, stays in the EU and is never used for training. Batches benefit too, when many lines share the same prefix.

Prompt caching: reuse long prompts across requests

How prompt caching works

Structuring prompts for cache hits

Measuring the effect

Limits and edge cases

Prompt caching: common questions

Related

Move your AI workloads to EU infrastructure