---
title: "Streaming responses with server-sent events · EU LLM API"
description: "Stream tokens from open models as they are generated, with OpenAI and Anthropic server-sent events. Examples, chunk format and error handling."
canonical: "https://inference.eu/api/streaming/"
language: "en"
---

Guides · Streaming

# Streaming LLM responses with server-sent events

Streaming sends the model's answer to you token by token while it is being generated, instead of after it is finished. Users see text straight away, long answers feel faster, and you can stop generation early. The inference.eu API streams in the OpenAI format for chat completions and Responses, and in the Anthropic format for Messages.

[Contact sales](https://inference.eu/contact/)[Example request](#example)

1.  01API format: OpenAI and Anthropic
2.  02EU data centres
3.  03Covered by one DPA

stream.py

```
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.inference.eu/v1", api_key=os.environ["INFERENCE_EU_API_KEY"])

stream = client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[{"role": "user", "content": "Explain the difference between a GmbH and an AG in five sentences."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n\n{chunk.usage.total_tokens} tokens")
```

## How streaming works

Set `stream` to `true` in the request. The API keeps the HTTP connection open and sends server-sent events: lines that start with `data:` followed by a JSON object. For chat completions, each object is a `chat.completion.chunk` whose `choices[0].delta` holds the new piece of text in `content`. The stream ends with the line `data: [DONE]`.

The official SDKs handle the parsing. In Python you iterate over the returned stream, in JavaScript you use `for await`. If you call the API from your own HTTP client, read the response body line by line, ignore empty lines and comments, and parse each `data:` payload as JSON.

## Usage, finish reasons and tool calls

By default, streamed chunks do not include token usage. Add `stream_options: {"include_usage": true}` and the last chunk before `[DONE]` carries a `usage` object with an empty `choices` array, which is why the example checks `chunk.choices` before reading it.

The chunk that ends the answer has a `finish_reason`, such as `stop` or `length`. Tool calls stream too: `delta.tool_calls` arrives in pieces, with the function name first and the JSON arguments spread across several chunks. Concatenate the argument strings by `index` and only parse them once `finish_reason` is `tool_calls`. Reasoning models may stream their thinking in a separate field before the answer.

## Responses and Anthropic Messages events

The Responses API streams typed events instead of chunks. Text arrives in `response.output_text.delta` events, tool calls and other output items have their own events, and `response.completed` carries the final response object with usage. Listen for the event types you need and ignore the rest.

The Messages endpoint uses Anthropic's sequence: `message_start`, then `content_block_start`, a series of `content_block_delta` events with `text_delta` or `input_json_delta`, `content_block_stop`, a `message_delta` with `stop_reason` and usage, and finally `message_stop`. The Anthropic SDKs provide helpers such as `text_stream` that yield only the text.

## Errors, timeouts and proxies

Errors that happen before generation starts, such as a bad key or an unknown model, return a normal HTTP error and no stream. Errors during generation arrive as an error event inside the stream, after which the connection closes; your loop should handle that and decide whether to retry.

Proxies and load balancers sometimes buffer responses, which turns a stream into one late block of text. Disable buffering for these routes, for example with `X-Accel-Buffering: no` on nginx, and set idle timeouts longer than the gap between tokens on slow reasoning models. If a user closes the page, cancel the request so generation stops.

## Streaming

Category

Guides

API format

OpenAI and Anthropic

Base URL

api.inference.eu/v1

Processing

EU data centres

Training on your data

Never

## Streaming: common questions

The questions buyers, developers and legal teams ask us most. Something missing? Ask us directly.

Set \`stream\` to \`true\` in a chat completions, Responses or Messages request. The API returns server-sent events as the model generates. With the OpenAI or Anthropic SDK, iterate over the returned stream object and print or forward each text delta as it arrives.

For chat completions, add \`stream\_options: {"include\_usage": true}\`. The final chunk before \`data: \[DONE\]\` then contains a \`usage\` object with prompt and completion tokens and an empty \`choices\` list. The Responses and Messages formats include usage in their final events automatically.

Yes. Tool call names and arguments arrive in pieces in \`delta.tool\_calls\`. Collect the argument fragments for each tool call \`index\`, join them into one string, and parse the JSON when \`finish\_reason\` is \`tool\_calls\`. Then run your function and send the result back as usual.

The model generates the same tokens either way, so the total work is the same. Streaming changes when you receive them: the first words appear almost immediately instead of after the full answer. Billing is based on tokens, not on whether you streamed.

Something between the API and your user is buffering the response, usually a reverse proxy, a serverless platform or a compression middleware. Turn off buffering and compression for the streaming route, flush after every write, and make sure your own server forwards each event as soon as it receives it.

Close the connection or break out of the loop and close the stream object. Generation stops shortly afterwards, and you are billed for tokens generated up to that point. In a web app, cancel the upstream request when the user navigates away or presses stop.

Do not call the API from the browser with your secret key. Stream from your server instead and forward the events to the browser using server-sent events, a streamed fetch response or WebSockets. Frameworks such as the Vercel AI SDK do this for you.

Yes, with Anthropic's event names. Requests to \`/v1/messages\` with \`stream: true\` return \`message\_start\`, \`content\_block\_delta\` and the other standard events, so the Anthropic SDKs and tools such as Claude Code stream exactly as they do elsewhere.

[Ask a question](https://inference.eu/contact/)

## Related

[API overview](https://inference.eu/api/)

-   [Chat completionsSend chat messages to open-weight models with the OpenAI chat completions format. Requests are processed in EU data centres and never used for training.Endpoints](https://inference.eu/api/chat-completions/)
-   [ResponsesUse the OpenAI Responses API format with open-weight models on inference.eu. Typed output items, built-in tools and multi-turn state, processed in the EU.Endpoints](https://inference.eu/api/responses/)
-   [Messages (Anthropic format)Call open-weight models with the Anthropic Messages format on inference.eu. Works with Anthropic SDKs and Claude Code, processed in EU data centres.Endpoints](https://inference.eu/api/messages/)
-   [Tool callingLet open models call your functions and APIs with OpenAI-compatible tool calling. Define tools, run the loop, handle arguments, processed in the EU.Guides](https://inference.eu/api/tool-calling/)

## Move your AI workloads to EU infrastructure

Tell us which models you use and roughly how much traffic you have. We reply within one working day with pricing and next steps.

-   Reply within one working day
-   Pricing for your volume
-   DPA ready to sign