--- title: "Prompt caching for LLM APIs · faster repeated prompts in the EU" description: "Reuse the processed start of a prompt across requests with automatic prompt caching. Structure prompts for cache hits and read cached_tokens in usage." canonical: "https://inference.eu/api/prompt-caching/" language: "en" --- Guides · Prompt caching # Prompt caching: reuse long prompts across requests Many applications send the same long text at the start of every request: a system prompt, tool definitions, a policy document or the earlier turns of a conversation. Prompt caching lets the model reuse the work it already did on that shared start, so repeated requests begin answering sooner. It works automatically, and you can see in every response how much of the prompt came from the cache. [Contact sales](https://inference.eu/contact/)[Example request](#example) 1. 01API format: OpenAI and Anthropic 2. 02EU data centres 3. 03Covered by one DPA cached\_prompt.py ``` import os from openai import OpenAI client = OpenAI(base_url="https://api.inference.eu/v1", api_key=os.environ["INFERENCE_EU_API_KEY"]) # Long, stable prefix first: policy text that is the same for every question POLICY = open("travel-expenses-policy.md").read() def ask(question: str): reply = client.chat.completions.create( model="gpt-oss-120b", messages=[ {"role": "system", "content": "Answer questions about the travel policy below. Quote the section number.\n\n" + POLICY}, {"role": "user", "content": question}, # variable part last ], ) cached = reply.usage.prompt_tokens_details.cached_tokens print(f"{cached} of {reply.usage.prompt_tokens} prompt tokens came from the cache") return reply.choices[0].message.content ask("Can I book business class for a six-hour flight?") ask("What is the daily meal allowance in Germany?") ``` ## How prompt caching works Before a model writes anything, it processes the whole prompt. When a new request begins with exactly the same tokens as a recent one, the API can reuse the processed state for that shared prefix and only process what is new. This is called prefix caching: it matches from the first token onwards and stops at the first difference. Caching is automatic for supported models on the chat completions, Responses and Messages endpoints, with no extra fields needed. The cache lives in memory in the same European data centres that process your requests, is kept separate between organisations, and expires after a period without use. ## Structuring prompts for cache hits Put what stays the same first and what changes last. A good order is: system prompt, tool definitions, long reference documents, earlier conversation turns, and finally the new user message. Even a small change near the start, such as a timestamp or user name in the system prompt, makes everything after it a miss. Keep the stable part byte-for-byte identical: the same whitespace, the same order of tools, the same JSON serialisation. In chat applications, append new turns rather than rewriting history, so each request extends the cached prefix of the one before. Send related requests close together in time while the cache is still warm. ## Measuring the effect Every response reports how many prompt tokens came from the cache. In chat completions read `usage.prompt_tokens_details.cached_tokens`, in the Responses API `usage.input_tokens_details.cached_tokens`, and in the Messages format `usage.cache_read_input_tokens`. Log this alongside total prompt tokens to see your hit rate per feature. A low hit rate usually means something at the start of the prompt changes on every call. Compare two consecutive prompts and look for the first difference. Anthropic-format clients that send `cache_control` markers on content blocks work unchanged; the markers are accepted. How cached tokens are reflected on your invoice depends on your plan, and we confirm it with you. ## Limits and edge cases Very short prompts are not worth caching, so caching only applies above a minimum prompt length that depends on the model. Cache entries are evicted after a period of inactivity and under heavy load, so a hit is likely but never guaranteed; your application must work the same either way. Caching does not change the output. The model reads the same tokens and produces the same kind of answer; only the work on the shared prefix is skipped. Cached data is used only to serve your organisation's requests, stays in the EU and is never used for training. Batches benefit too, when many lines share the same prefix. ## Prompt caching Category Guides API format OpenAI and Anthropic Base URL api.inference.eu/v1 Processing EU data centres Training on your data Never ## Prompt caching: common questions The questions buyers, developers and legal teams ask us most. Something missing? Ask us directly. Prompt caching reuses the processed form of a prompt's opening tokens when a new request starts with the same text. The model then only processes the new part. Long system prompts, tool definitions and documents that repeat across requests get processed once and reused while the cache is warm. No. Caching is automatic for supported models. You get the most from it by putting stable content at the start of your prompt and variable content at the end. Each response shows how many tokens came from the cache, so you can check that it works. Read \`usage.prompt\_tokens\_details.cached\_tokens\` in chat completions, \`usage.input\_tokens\_details.cached\_tokens\` in the Responses API, or \`usage.cache\_read\_input\_tokens\` in the Messages format. A value above zero means part of the prompt came from the cache; zero means the whole prompt was processed afresh. Usually something near the start of the prompt changes on every request, such as the current date, a request id or a user name in the system prompt. Move such details to the end. Also check that the prompt is long enough to qualify and that requests are not too far apart in time. No. The model reads exactly the same tokens with or without the cache. Caching only skips repeating work on the shared opening of the prompt, so answers are the same kind you would get without it. No. Cache entries are kept separate per organisation and are only used to serve your own requests. They are held in memory in European data centres, expire after a period without use, and are never used to train models. Yes. Requests to \`/v1/messages\` benefit from automatic caching, and clients that add \`cache\_control\` to content blocks, as many Anthropic tools do, work without changes. Cache usage is reported in \`cache\_read\_input\_tokens\`, as Anthropic clients expect. How cached tokens are billed depends on your plan, and we confirm it with you before you go live. Regardless of billing, caching makes long repeated prompts quicker to start answering, which is noticeable in chat applications and agent loops. [Ask a question](https://inference.eu/contact/) ## Related [API overview](https://inference.eu/api/) - [Chat completionsSend chat messages to open-weight models with the OpenAI chat completions format. Requests are processed in EU data centres and never used for training.Endpoints](https://inference.eu/api/chat-completions/) - [Count tokensCount the input tokens of a Messages request before you send it. Anthropic-compatible count\_tokens endpoint on inference.eu, processed in the EU.Endpoints](https://inference.eu/api/count-tokens/) - [Messages (Anthropic format)Call open-weight models with the Anthropic Messages format on inference.eu. Works with Anthropic SDKs and Claude Code, processed in EU data centres.Endpoints](https://inference.eu/api/messages/) - [Rate limits and errorsHow rate limits and errors work on the inference.eu API. HTTP status codes, the error object, rate-limit headers and a retry pattern that holds up.Guides](https://inference.eu/api/rate-limits-and-errors/) ## Move your AI workloads to EU infrastructure Tell us which models you use and roughly how much traffic you have. We reply within one working day with pricing and next steps. - Reply within one working day - Pricing for your volume - DPA ready to sign