Streaming LLM responses with server-sent events

Set stream to true in the request. The API keeps the HTTP connection open and sends server-sent events: lines that start with data: followed by a JSON object. For chat completions, each object is a chat.completion.chunk whose choices[0].delta holds the new piece of text in content. The stream ends with the line data: [DONE].

The official SDKs handle the parsing. In Python you iterate over the returned stream, in JavaScript you use for await. If you call the API from your own HTTP client, read the response body line by line, ignore empty lines and comments, and parse each data: payload as JSON.

By default, streamed chunks do not include token usage. Add stream_options: {"include_usage": true} and the last chunk before [DONE] carries a usage object with an empty choices array, which is why the example checks chunk.choices before reading it.

The chunk that ends the answer has a finish_reason, such as stop or length. Tool calls stream too: delta.tool_calls arrives in pieces, with the function name first and the JSON arguments spread across several chunks. Concatenate the argument strings by index and only parse them once finish_reason is tool_calls. Reasoning models may stream their thinking in a separate field before the answer.

The Responses API streams typed events instead of chunks. Text arrives in response.output_text.delta events, tool calls and other output items have their own events, and response.completed carries the final response object with usage. Listen for the event types you need and ignore the rest.

The Messages endpoint uses Anthropic's sequence: message_start, then content_block_start, a series of content_block_delta events with text_delta or input_json_delta, content_block_stop, a message_delta with stop_reason and usage, and finally message_stop. The Anthropic SDKs provide helpers such as text_stream that yield only the text.

Errors that happen before generation starts, such as a bad key or an unknown model, return a normal HTTP error and no stream. Errors during generation arrive as an error event inside the stream, after which the connection closes; your loop should handle that and decide whether to retry.

Proxies and load balancers sometimes buffer responses, which turns a stream into one late block of text. Disable buffering for these routes, for example with X-Accel-Buffering: no on nginx, and set idle timeouts longer than the gap between tokens on slow reasoning models. If a user closes the page, cancel the request so generation stops.