Stream responses

Streaming sends the reply while it is generated, so a reader sees the first words after the time to the first token rather than after the whole reply. Mortar streams with server-sent events, in the same chat.completion.chunk format as the OpenAI API.

Stream with an SDK

Set stream to true. To get the reply's token counts as well, set stream_options.include_usage: the last chunk before the stream ends then carries usage and no choices.

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.mortar.one/v1",
    api_key=os.environ["MORTAR_API_KEY"],
)

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    delta = chunk.choices[0].delta if chunk.choices else None
    if delta and delta.content:
        print(delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n{chunk.usage.total_tokens} tokens")

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.mortar.one/v1",
  apiKey: process.env.MORTAR_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "Qwen/Qwen2.5-7B-Instruct",
  messages: [{ role: "user", content: "Write a haiku about GPUs." }],
  stream: true,
  stream_options: { include_usage: true },
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.log(`\n${chunk.usage.total_tokens} tokens`);
}

curl -N https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Write a haiku about GPUs."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Read the stream yourself

Without an SDK, read the response body as text and split it into events at each blank line. In each event:

• A line starting with data: holds a chunk as JSON, or [DONE].
• A line starting with : is a keep-alive comment. Ignore it.
• [DONE] is the last event of a complete reply.

This Node.js 18 example prints the reply and fails loudly if the stream ends early:

const response = await fetch("https://api.mortar.one/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.MORTAR_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    messages: [{ role: "user", content: "Write a haiku about GPUs." }],
    stream: true,
  }),
});
if (!response.ok) throw new Error((await response.json()).error.message);

const decoder = new TextDecoder();
let buffer = "";
let finished = false;
for await (const bytes of response.body) {
  buffer += decoder.decode(bytes, { stream: true });
  let end;
  while ((end = buffer.indexOf("\n\n")) !== -1) {
    const event = buffer.slice(0, end);
    buffer = buffer.slice(end + 2);
    for (const line of event.split("\n")) {
      if (!line.startsWith("data: ")) continue;
      const data = line.slice("data: ".length);
      if (data === "[DONE]") {
        finished = true;
        continue;
      }
      const chunk = JSON.parse(data);
      if (chunk.error) throw new Error(chunk.error.message);
      process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
    }
  }
}
if (!finished) throw new Error("The stream ended before [DONE], so the reply is incomplete.");

Know when a reply is complete

A reply is complete only when [DONE] arrives. Before it, the finish chunk's finish_reason says why generation ended:

• stop: the model finished its answer.
• length: the reply hit max_tokens or filled the context window. Raise max_tokens, or shorten the conversation.

If the model server fails mid-reply, Mortar sends one event holding an error envelope whose type is upstream_error, then closes the stream without [DONE]. The text you received is a fragment: show it as failed, not as an answer.

Cancel a generation

Close the connection and Mortar stops generating. In Python call stream.close(); in TypeScript call stream.controller.abort(). Tokens generated before the stop are still counted.

When the process that wants to cancel isn't the one reading the stream — a cancel button handled by another server, for example — stop the generation by its ID, with a key from the same organization. Every chunk carries the completion's id, and the response's X-Mortar-Request-Id header holds it too:

curl -X POST https://api.mortar.one/v1/generations/chatcmpl-1e3a5c7b9d0f2a4c6e8b0d1f/stop \
  -H "Authorization: Bearer $MORTAR_API_KEY"

The stream then ends without a finish chunk; the usage chunk, if you asked for one, and [DONE] still follow. See Stop a generation.

Proxies and timeouts

Mortar sends Cache-Control: no-cache, no-transform and X-Accel-Buffering: no, which stop well-behaved proxies and CDNs from buffering the stream. If you run your own proxy in front of Mortar, turn off response buffering for /v1/chat/completions too, or readers will see the reply arrive all at once.

A stream stays open as long as the reply takes; Mortar puts no time limit on it. When the deployment forwards to a model server, Mortar also sends a : ping comment every 10 seconds, so an idle-connection timeout of 30 seconds or more keeps the stream open even while the model is slow to start.