# Stream responses

> Stream chat completions from Mortar as server-sent events with the OpenAI Python and TypeScript SDKs or plain HTTP — read chunks, get token usage, detect incomplete replies and cancel a generation.

Streaming sends the reply while it is generated, so a reader sees the first words after the time to the first token rather than after the whole reply. Mortar streams with server-sent events, in the same `chat.completion.chunk` format as the OpenAI API.

## Stream with an SDK

Set `stream` to true. To get the reply's token counts as well, set `stream_options.include_usage`: the last chunk before the stream ends then carries `usage` and no choices.

```python tab="Python" group="Stream with an SDK"
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.mortar.one/v1",
    api_key=os.environ["MORTAR_API_KEY"],
)

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    delta = chunk.choices[0].delta if chunk.choices else None
    if delta and delta.content:
        print(delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n{chunk.usage.total_tokens} tokens")
```

```typescript tab="TypeScript"
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.mortar.one/v1",
  apiKey: process.env.MORTAR_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "Qwen/Qwen2.5-7B-Instruct",
  messages: [{ role: "user", content: "Write a haiku about GPUs." }],
  stream: true,
  stream_options: { include_usage: true },
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.log(`\n${chunk.usage.total_tokens} tokens`);
}
```

```bash tab="curl"
curl -N https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Write a haiku about GPUs."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'
```

> [!TIP]
> Check that a chunk has choices before reading them. The usage chunk has an empty `choices` array, and the finish chunk has no `delta`.

## Read the stream yourself

Without an SDK, read the response body as text and split it into events at each blank line. In each event:

- A line starting with `data: ` holds a chunk as JSON, or `[DONE]`.
- A line starting with `:` is a keep-alive comment. Ignore it.
- `[DONE]` is the last event of a complete reply.

This Node.js 18 example prints the reply and fails loudly if the stream ends early:

```javascript title="stream.mjs"
const response = await fetch("https://api.mortar.one/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.MORTAR_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    messages: [{ role: "user", content: "Write a haiku about GPUs." }],
    stream: true,
  }),
});
if (!response.ok) throw new Error((await response.json()).error.message);

const decoder = new TextDecoder();
let buffer = "";
let finished = false;
for await (const bytes of response.body) {
  buffer += decoder.decode(bytes, { stream: true });
  let end;
  while ((end = buffer.indexOf("\n\n")) !== -1) {
    const event = buffer.slice(0, end);
    buffer = buffer.slice(end + 2);
    for (const line of event.split("\n")) {
      if (!line.startsWith("data: ")) continue;
      const data = line.slice("data: ".length);
      if (data === "[DONE]") {
        finished = true;
        continue;
      }
      const chunk = JSON.parse(data);
      if (chunk.error) throw new Error(chunk.error.message);
      process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
    }
  }
}
if (!finished) throw new Error("The stream ended before [DONE], so the reply is incomplete.");
```

## Know when a reply is complete

A reply is complete only when `[DONE]` arrives. Before it, the finish chunk's `finish_reason` says why generation ended:

- `stop`: the model finished its answer.
- `length`: the reply hit `max_tokens` or filled the context window. Raise `max_tokens`, or shorten the conversation.

If the model server fails mid-reply, Mortar sends one event holding an [error envelope](/docs/api#errors) whose `type` is `upstream_error`, then closes the stream without `[DONE]`. The text you received is a fragment: show it as failed, not as an answer.

## Cancel a generation

Close the connection and Mortar stops generating. In Python call `stream.close()`; in TypeScript call `stream.controller.abort()`. Tokens generated before the stop are still counted.

When the process that wants to cancel isn't the one reading the stream — a cancel button handled by another server, for example — stop the generation by its ID, with a key from the same organization. Every chunk carries the completion's `id`, and the response's `X-Mortar-Request-Id` header holds it too:

```bash title="Request" status="200 404"
curl -X POST https://api.mortar.one/v1/generations/chatcmpl-1e3a5c7b9d0f2a4c6e8b0d1f/stop \
  -H "Authorization: Bearer $MORTAR_API_KEY"
```

The stream then ends without a finish chunk; the usage chunk, if you asked for one, and `[DONE]` still follow. See [Stop a generation](/docs/api/chat-completions#stop-a-generation).

## Proxies and timeouts

Mortar sends `Cache-Control: no-cache, no-transform` and `X-Accel-Buffering: no`, which stop well-behaved proxies and CDNs from buffering the stream. If you run your own proxy in front of Mortar, turn off response buffering for `/v1/chat/completions` too, or readers will see the reply arrive all at once.

A stream stays open as long as the reply takes; Mortar puts no time limit on it. When the deployment forwards to a model server, Mortar also sends a `: ping` comment every 10 seconds, so an idle-connection timeout of 30 seconds or more keeps the stream open even while the model is slow to start.