Docs
Models Sign in

Chat completions

Last updated View as Markdown Send feedback

On this page
  1. Create a chat completion
    1. Request body
    2. Request headers
    3. Response headers
    4. Example
    5. The chat completion object
  2. Streaming
    1. The chunk object
    2. Usage in a stream
    3. When a stream ends early
    4. Errors
  3. Call tools
  4. Stop a generation
    1. Path parameters
    2. Example
    3. Response

Chat completions generate the next assistant message in a conversation. The endpoint follows the OpenAI chat-completions API, so the OpenAI SDKs work against it; the compatibility notes list the differences.

Create a chat completion

POST /v1/chat/completions

Send the conversation so far and receive the assistant's reply, either as one JSON object or as a stream of events. Needs an API key.

Request body

Field Type Required Description
model string No The model to use. Defaults to Qwen/Qwen2.5-7B-Instruct. The model catalog lists the others.
messages array Yes The conversation, oldest message first. Between 1 and 512 messages.
messages[].role string Yes Who wrote the message: system, user, assistant or tool.
messages[].content string, array or null Yes The message text. An array of content parts is accepted: the text of each part is joined with line breaks, and parts without text, such as images, are dropped. Only an assistant message with tool_calls may have null content, or leave it out.
messages[].tool_calls array No The calls an assistant message made, as the model returned them. Up to 128.
messages[].tool_calls[].id string Yes The call's ID. The tool message that answers it repeats it in tool_call_id.
messages[].tool_calls[].type string No function.
messages[].tool_calls[].function.name string Yes The function that was called.
messages[].tool_calls[].function.arguments string Yes Its arguments, as JSON text.
messages[].tool_call_id string On tool messages The ID of the call a tool message answers. Its content is what the function returned.
stream boolean No Whether to stream the reply as server-sent events. Defaults to true, unlike the OpenAI API.
stream_options.include_usage boolean No When streaming, send one more chunk with token usage before the stream ends. Defaults to false.
max_tokens integer No The most tokens to generate. Must be a positive whole number. Defaults to 1,024, and is lowered when the context window has less room left. A deployment that forwards to a model server passes it on, and the model server ends the reply there.
user string No Names the session this request belongs to when X-Mortar-Session isn't sent. See cache-friendly prompts.
tools array No Functions the model may call, up to 128. See call tools.
tools[].type string Yes function.
tools[].function.name string Yes 1 to 64 letters, digits, underscores or dashes, different for each tool.
tools[].function.description string No What the function does, for the model to read.
tools[].function.parameters object No A JSON Schema object describing the arguments.
tools[].function.strict boolean No Asks a model server that supports it to follow parameters exactly.
tool_choice string or object No auto, the default when there are tools, lets the model decide. none stops it calling any, required makes it call at least one, and {"type": "function", "function": {"name": "get_weather"}} makes it call that one. Needs tools.
parallel_tool_calls boolean No Whether the model may make more than one call in a reply. Needs tools.

Mortar ignores any other field, such as temperature.

The prompt, the reply and 32 tokens of chat-template overhead must fit in the model's context window — 4,096 tokens for Qwen/Qwen2.5-7B-Instruct on this deployment, as reported by GET /v1/models. If fewer than 64 tokens would be left for the reply, the request fails with context_length_exceeded.

Request headers

Header Required Description
Authorization Yes Bearer followed by your API key.
Content-Type Yes application/json.
X-Mortar-Session No Names the conversation within your organization: up to 128 letters, digits, -, _, . or :, and any other character is dropped. Requests in one session are routed to the replica that already holds their prompt prefix. Defaults to user; with neither, the request starts a session of its own.
X-Mortar-Optimize No off processes the prompt exactly as sent. Opt-in values include json-tool-blocks for JSON-fenced data-tool results and collapse-spaces for aligned tool output. on asks for the lossless defaults only, overriding an organization default of off. Any unknown value is refused with 400. The deployment's mode still applies: in shadow mode, the default, nothing is changed, only measured, and when it is off no value turns optimization on. Keep the value the same for a whole conversation.
X-Mortar-Tool-Select No named keeps only the function explicitly named by object-form tool_choice. This is an opt-in progressive-disclosure fast path for clients that send a full catalogue even though they have already selected one function. Other values, or named without a named choice, are refused with 400.
X-Mortar-Dedupe No in-flight coalesces concurrently executing, byte-identical non-streaming requests within the same tenant. It requires stream: false and an Idempotency-Key. A key reused concurrently with a different request is refused with 409; no completed response is cached.
Idempotency-Key With in-flight deduplication A 1–128 character operation key using letters, digits, ., _, - or :. It is scoped to the authenticated tenant and combined with the exact request and optimization/session headers.
X-Request-Id No Your own ID for the request, echoed back in the response.

Response headers

Header Description
X-Mortar-Request-Id The completion's ID, the same as id in the body. Pass it to stop the generation.
X-Mortar-Deployment The deployment that served the request.
X-Mortar-Model The model that generated the reply.
X-Mortar-Upstream Present when the deployment forwards requests to a model server, and names that server.
X-Mortar-Session The session the request ran in: the one you named, or the one it started. Follow it on the telemetry stream.
X-Mortar-Optimizer What the prompt optimizer did: a report such as ruleset=2; mode=on; bytes=541; saved=77; passes=json-compact/1,json-tool-blocks/1; elapsed_us=212, where elapsed_us is how long optimization took in microseconds, or ruleset=2; mode=off while it's off. In shadow mode, the default, the prompt was used as sent and saved is what optimization would have removed.
X-Mortar-Tool-Selector Present when named tool selection ran, for example mode=named; tools=20->1; tokens=2860->144; saved=2716. Counts use the same tokenizer as prompt planning.
X-Mortar-Dedupe leader for the request that reached inference, or shared for a concurrent request that reused that exact in-flight result. Shared responses intentionally carry the leader's completion ID.
X-Request-Id Your X-Request-Id, or one Mortar generated.

Example

Requestbash
curl https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a concise assistant for an inference platform."},
      {"role": "user", "content": "What does max_tokens limit?"}
    ],
    "max_tokens": 200,
    "stream": false
  }'
Responsejson
{
  "id": "chatcmpl-9b2f4d6e8a0c4b1d3f5e7a9c",
  "object": "chat.completion",
  "created": 1767225600,
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "It caps how many tokens the model may generate for this reply."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 32,
    "completion_tokens": 13,
    "total_tokens": 45
  }
}

The chat completion object

Field Type Description
id string chatcmpl- followed by 24 hexadecimal characters.
object string Always chat.completion.
created integer When the request was accepted, in Unix seconds.
model string The model that generated the reply.
choices array Always exactly one choice.
choices[].index integer Always 0.
choices[].message.role string Always assistant.
choices[].message.content string or null The reply, or null when the reply only calls tools.
choices[].message.tool_calls array Present when the model calls tools: the calls, in order.
choices[].message.tool_calls[].id string The call's ID, to repeat in the tool_call_id of the message that answers it.
choices[].message.tool_calls[].type string Always function.
choices[].message.tool_calls[].function.name string The function to call.
choices[].message.tool_calls[].function.arguments string Its arguments, as JSON text. Parse them before you trust them: a model can produce invalid JSON.
choices[].logprobs null Always null. Log probabilities aren't returned.
choices[].finish_reason string stop when the reply is complete, tool_calls when it calls tools, and length when it was cut off by max_tokens or the context window. When the deployment forwards to a model server, that server's own reason is passed through. A server that ignores max_tokens is stopped by Mortar at three times it plus 64 tokens, by Mortar's estimate, with length.
usage.prompt_tokens integer Tokens in the prompt, including chat-template framing.
usage.completion_tokens integer Tokens in the reply.
usage.total_tokens integer The sum of the two.

When the deployment forwards to a model server that reports usage, these are that server's own counts, and the prompt_tokens_details and completion_tokens_details it sends, such as cached_tokens, are passed through unchanged. Otherwise, and for a reply Mortar ended itself because the model server ran on past max_tokens, they're Mortar's own estimate, the same one its context-window check uses. Either way prompt_tokens counts the prompt after the prompt optimizer, which is the prompt the model reads, and the telemetry stream says in usageSource which kind of count the request was billed on.

Streaming

With stream true — or left out — the response is a text/event-stream. Each event is one data: line holding a chat.completion.chunk, followed by a blank line, and the stream ends with data: [DONE].

This request leaves stream out, so the reply streams:

Requestbash
curl -N https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What does max_tokens limit?"}],
    "max_tokens": 200
  }'
Response
data: {"id":"chatcmpl-1e3a5c7b9d0f2a4c6e8b0d1f","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"delta":{"role":"assistant"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1e3a5c7b9d0f2a4c6e8b0d1f","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"delta":{"content":"It caps how"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1e3a5c7b9d0f2a4c6e8b0d1f","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"logprobs":null,"finish_reason":"stop"}]}

data: [DONE]

The chunks arrive in this order:

  1. One chunk whose delta holds role: "assistant" and no text.
  2. One chunk per piece of the reply, each with the next text in delta.content.
  3. One chunk with finish_reason set and no delta.
  4. With stream_options.include_usage, one chunk with an empty choices array and a usage object.
  5. data: [DONE].

The chunk object

Field Type Description
id string The completion's ID, the same on every chunk.
object string Always chat.completion.chunk.
created integer When the request was accepted, in Unix seconds.
model string The model generating the reply.
choices array One choice, or none on the usage chunk.
choices[].index integer Always 0.
choices[].delta.role string assistant, on the first chunk only.
choices[].delta.content string The next piece of the reply.
choices[].delta.tool_calls array The next piece of the tool calls, when the model calls tools.
choices[].delta.tool_calls[].index integer Which call the piece belongs to, counting from 0. Pieces of parallel calls can interleave.
choices[].delta.tool_calls[].id string The call's ID, in its first piece.
choices[].delta.tool_calls[].type string function, in its first piece.
choices[].delta.tool_calls[].function.name string The function, in its first piece.
choices[].delta.tool_calls[].function.arguments string The next piece of the arguments. Join a call's pieces in order to get its JSON.
choices[].logprobs null Always null.
choices[].finish_reason string or null null on every chunk except the finish chunk, which carries stop, tool_calls or length.
usage object Only on the usage chunk: prompt_tokens, completion_tokens and total_tokens, counted as for a chat completion.

Usage in a stream

Ask for usage and Mortar adds one chunk after the finish chunk:

Requestbash
curl -N https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What does max_tokens limit?"}],
    "max_tokens": 200,
    "stream": true,
    "stream_options": {"include_usage": true}
  }'
Response, last three events
data: {"id":"chatcmpl-7c9e1a3b5d0f2c4e6a8b0d2f","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"logprobs":null,"finish_reason":"stop"}]}

data: {"id":"chatcmpl-7c9e1a3b5d0f2c4e6a8b0d2f","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[],"usage":{"prompt_tokens":17,"completion_tokens":13,"total_tokens":30}}

data: [DONE]

When a stream ends early

  • A model-server error mid-stream arrives as one event holding the error envelope with type set to upstream_error, and the stream closes without [DONE]. A stream that closes without [DONE] is incomplete; treat it as failed.
  • A stopped generation sends no finish chunk. The usage chunk, if you asked for it, and [DONE] still follow.
  • Keep-alive comments. When the deployment forwards to a model server, Mortar sends a : ping comment line every 10 seconds so proxies don't close a quiet connection. Event-stream parsers ignore comments.

The streaming guide shows how to consume the stream in each SDK.

Errors

Status Code When
400 The body isn't valid JSON; messages is missing, empty or longer than 512 entries; a message lacks a string role or a string-or-array content; a tool call, a tool message's tool_call_id, tools, tool_choice or parallel_tool_calls isn't shaped as described above; max_tokens isn't a positive whole number; or X-Mortar-Optimize has a value Mortar doesn't recognise.
400 context_length_exceeded Fewer than 64 tokens of the context window are left for the reply. Start a new conversation or send fewer messages.
404 model_not_found model isn't in the catalog.
503 model_not_available model is in the catalog but not deployed yet.
502, 503, 504 upstream_error, upstream_unavailable, upstream_timeout The deployment forwards to a model server, and that server failed, was unreachable or timed out.
Request for an unknown modelbash
curl https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}], "stream": false}'
Responsejson
{
  "error": {
    "message": "The model `gpt-4o` does not exist.",
    "type": "invalid_request_error",
    "code": "model_not_found"
  }
}

The API overview lists the errors every endpoint can return.

Call tools

Describe functions in tools and the model can answer with calls to them instead of text. The reply then has finish_reason tool_calls, null content, and the calls in message.tool_calls. Run them, and send the conversation back with that assistant message as you received it, followed by one tool message for each call: its tool_call_id names the call and its content holds what the function returned. The model reads the results and answers, or calls again.

Requestbash
curl https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "stream": false
  }'
Responsejson
{
  "id": "chatcmpl-4d6f8a0c2e4b6d8f0a2c4e6b",
  "object": "chat.completion",
  "created": 1767225600,
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_8e1f3a5c7b9d2e4f6a8c0b1d",
            "type": "function",
            "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}
          }
        ]
      },
      "logprobs": null,
      "finish_reason": "tool_calls"
    }
  ],
  "usage": {
    "prompt_tokens": 61,
    "completion_tokens": 9,
    "total_tokens": 70
  }
}

Send the result back with the call that asked for it:

Requestbash
curl https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "What is the weather in Paris?"},
      {"role": "assistant", "content": null, "tool_calls": [{
        "id": "call_8e1f3a5c7b9d2e4f6a8c0b1d",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}
      }]},
      {"role": "tool", "tool_call_id": "call_8e1f3a5c7b9d2e4f6a8c0b1d", "content": "{\"temp_c\": 21, \"sky\": \"clear\"}"}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "stream": false
  }'
Response, abbreviatedjson
{
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "It's 21 °C and clear in Paris."},
      "logprobs": null,
      "finish_reason": "stop"
    }
  ]
}

Streamed, each call arrives in pieces. The first piece of a call carries its index, id, type and function name; the pieces after it carry only the index and the next part of the arguments. Join each call's arguments in order. With parallel_tool_calls, the pieces of several calls can interleave, and index tells them apart.

Requestbash
curl -N https://api.mortar.one/v1/chat/completions \
  -H "Authorization: Bearer $MORTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
      }
    }],
    "tool_choice": {"type": "function", "function": {"name": "get_weather"}}
  }'
Response
data: {"id":"chatcmpl-6b8d0f2a4c6e8a0b2d4f6a8c","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"delta":{"role":"assistant"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-6b8d0f2a4c6e8a0b2d4f6a8c","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_2b4d6f8a0c1e3a5c7e9b1d3f","type":"function","function":{"name":"get_weather","arguments":""}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-6b8d0f2a4c6e8a0b2d4f6a8c","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\":\"Paris\"}"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-6b8d0f2a4c6e8a0b2d4f6a8c","object":"chat.completion.chunk","created":1767225600,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"logprobs":null,"finish_reason":"tool_calls"}]}

data: [DONE]

When the deployment forwards to a model server, the tools, tool_choice and parallel_tool_calls go to it as you sent them, and its calls come back as it made them, streamed pieces included. With no model server, Mortar's scripted model calls tools itself, so an agent loop can be tested without one. It calls the function tool_choice names, or every tool for required, or only the first when parallel_tool_calls is false. Otherwise it calls the first tool, until the conversation ends with tool results, which it answers in text. It fills each required argument with an empty value of its type.

Stop a generation

POST /v1/generations/{id}/stop

Stops a chat completion while it is still being generated. Closing the connection stops a generation too; use this endpoint when the request that is generating isn't yours to close, such as a cancel button handled by another server. Needs an API key from the organization that started the generation.

Path parameters

Parameter Type Description
id string The completion's ID, from id in any chunk or from the X-Mortar-Request-Id header.

Example

Requestbash
curl -X POST https://api.mortar.one/v1/generations/chatcmpl-1e3a5c7b9d0f2a4c6e8b0d1f/stop \
  -H "Authorization: Bearer $MORTAR_API_KEY"
Responsejson
{
  "stopped": true,
  "id": "chatcmpl-1e3a5c7b9d0f2a4c6e8b0d1f"
}

Response

Field Type Description
stopped boolean true if a running generation was stopped.
id string The ID you sent.

The status is 200 when a generation was stopped and 404 when none with that ID is running for your organization: it already finished, the ID is wrong, or another organization started it. Mortar answers all three the same way, so the endpoint can't be used to find out which IDs are live. Both statuses carry the same body; the 404 doesn't use the error envelope.