The fastest path to production
- Control plane and data plane both run in Mortar's cloud
- Mortar operates the stack and the GPU capacity
- Pick a model, set limits, get an endpoint
OpenAI-compatible LLM inference. Managed Cloud or BYOC.
Serve open-source and custom models with prefix-aware routing, tiered KV caching, autoscaling, GPU governance, and token-level cost visibility—inside Mortar's cloud or your own cloud account.
Mortar Console
Acme Corp
Deployments
4 deployments across 3 environments
| Deployment | Model | Environment | GPU | Replicas | Status |
|---|---|---|---|---|---|
| support-assistant | Qwen 2.5 7B Instruct | L4 | 3 / 6 | ||
| claims-summarizer | Llama 3.1 8B Instruct | A10G | 2 / 4 | ||
| code-review-bot | Qwen 2.5 32B Instruct | L40S | 1 / 3 | ||
| batch-classifier | Mistral 7B Instruct | L4 | 0 / 2 |
Rollout of code-review-bot to BYOC · acme-prod
- Deployment spec v7 publishedControl plane · 09:41
- Spec pulled by mortar-agentOutbound mTLS from acme-prod
- GPU node provisioned1× L40S in your AWS account
- Weights loadeds3://acme-models · customer-managed KMS key
- Readiness checks2 of 3 passing
- Shift trafficWaits for readiness
Private endpoint
https://llm.acme-prod.internal/v1
OpenAI-compatible. Prompts and outputs stay inside acme-prod's VPC. Only token counts, latency and node-hours reach Mortar.
Model catalog
Tested engine settings, quantization and GPU sizing for every model
Alibaba Qwen
- Size
- 7B dense
- Context
- 32K
- Precision
- BF16 · AWQ INT4
- GPU
- 1× L4 or A10G
Meta Llama
- Size
- 8B dense
- Context
- 128K
- Precision
- BF16 · FP8
- GPU
- 1× L4 or A10G
Mistral AI
- Size
- 7B dense
- Context
- 32K
- Precision
- BF16 · AWQ INT4
- GPU
- 1× L4 or A10G
Alibaba Qwen
- Size
- 32B dense
- Context
- 32K
- Precision
- BF16 · FP8
- GPU
- 1× L40S or 2× A10G
Meta Llama
- Size
- 70B dense
- Context
- 128K
- Precision
- BF16 · FP8
- GPU
- H100 class, multi-GPU
DeepSeek
- Size
- Mixture of Experts
- Context
- 128K
- Precision
- FP8
- GPU
- Multi-node H100/H200
Usage and cost
Last 30 days, all environments
- Requests
- 4.82MUp 12% vs prior 30 days
- Tokens processed
- 1.9BUp 18% vs prior 30 days
- Median time to first token
- 210 msDown 9% vs prior 30 days
- Blended cost per 1M tokens
- $1.53Down 6% vs prior 30 days
Spend by deployment
GPU quota by team
-
Support6 / 8 L4
6 of 8 L4 -
Claims4 / 4 A10G
4 of 4 A10G At quota
-
Research1 / 4 L40S
1 of 4 L40S 2 idle GPUs lent to batch jobs, reclaimed on demand
Product preview. Interface and data are illustrative.
Built on proven open-source infrastructure
- Kubernetes
- KServe
- vLLM
- SGLang
- llm-d
- LMCache
- KEDA
- Karpenter
- Kueue
- OpenCost
Choose Managed Cloud or Bring Your Own Cloud inference
BYOC changes where the data plane runs, not who runs the platform. Mortar builds, ships and operates the same software in both.
First available on Amazon EKS, with GKE and AKS to follow.
Manage distributed LLM data planes from one control plane
Mortar control plane
- Console
- Model catalog
- Policy
- Orchestration
- Usage and billing
Managed CloudMortar's cloud account
Mortar data plane
- Gateway
- Router
- Model runtime
- KV cache
- Autoscaling
- FinOps
mortar-agent
Your apps call /v1/chat/completions here
Bring Your Own CloudYour cloud account and VPC
Mortar data plane
- Gateway
- Router
- Model runtime
- KV cache
- Autoscaling
- FinOps
mortar-agent
Your apps call /v1/chat/completions here
Declare what you want
Choose a model, environment, GPU class, scaling range and budget and latency policy in one console.
Mortar reconciles it
An agent in each data plane pulls the versioned spec over an outbound, authenticated channel and applies it locally.
Call one API
Applications call an OpenAI-compatible endpoint served directly by the selected data plane.
Optimize routing, KV cache reuse, scaling, and token cost
-
Prefix-aware routing
Requests go to the replica that already holds their prompt prefix, so shared system prompts are not prefilled again.
One request to support-assistant: the replicas the router weighed, and what the one it chose already held. Sample data. Ready replica Waiting Cached support-assistant- 01 2 0 support-assistant- 02 Routed here 3 1,664 support-assistant- 03 0 0 - Reused from GPU HBM
- 1,664 tokens
- Prefilled
- 248 tokens
- Prompt
- 1,912 tokens
- Reason
- Prefix affinity. Replica already holds this conversation’s KV prefix.
- Time to first token
- 212 ms
-
Tiered KV cache
KV state is reused across GPU memory, host memory and storage tiers instead of being thrown away.
Where cached prefixes were found, and the cost of fetching a thousand tokens back. Sample data. Tier Hits Fetch per 1K GPU HBM 68% 0.4 ms Host memory 24% 3.1 ms Storage (LMCache) 8% 11.5 ms -
Token-level FinOps
Cost per million input and output tokens, broken down by model, project and tenant.
Thirty days of the preview’s deployments, priced per million tokens. Sample data. Deployment Tokens Per 1M support-assistant 820M $1.51 claims-summarizer 540M $1.59 code-review-bot 380M $1.61 batch-classifier 160M $1.19 Blended 1.9B $1.53 -
Queue-based autoscaling
Replicas scale on pending requests and cache pressure, and GPU nodes are added or removed to match.
-
Tenant isolation
Authenticated tenant identity, cache-key salting and per-tenant rate and budget limits at the gateway.
-
GPU fairness and quotas
Team quotas, borrowing and priority preemption keep interactive traffic fast while idle GPUs stay busy.
GPU quota by team, with idle capacity lent out. Sample data. -
Support6 of 8 L4
6 of 8 L4 -
Claims4 of 4 A10G
4 of 4 A10G At quota
-
Research1 of 4 L40S
1 of 4 L40S 2 idle GPUs lent to batch jobs, reclaimed on demand
-
-
Model choice
Curated open-source models with tested engine settings, or your own fine-tuned weights and LoRA adapters.
-
Policy and governance
Model allow-lists, environment placement, API-key lifecycle and change approval from one control plane.
Customer-controlled boundaries for BYOC inference
BYOC uses the same platform capabilities with infrastructure boundaries managed in your cloud account.
- Payload locality
- In BYOC, prompts, outputs, embeddings and model weights are processed in your environment; the architecture is designed to keep payloads out of the control plane.
- Outbound management
- The Mortar agent uses an authenticated mTLS connection to the control plane, avoiding a control-plane ingress path into your cluster.
- Your keys, your storage
- Keep models in your own storage under customer-managed encryption keys, read by narrow, auditable workload identities.
- Minimal telemetry
- Configure operational telemetry such as request and token counts, latency, and node-hours without sending request or response bodies.
- Local serving continuity
- The data-plane design keeps deployed models and the local gateway serving independently of routine control-plane communication.
- Audit-ready
- Evidence-ready logging and architecture documentation to support your security and compliance reviews.
{
"deployment": "claims-summarizer",
"environment": "acme-prod",
"window": "2026-10-28T09:00Z/PT1H",
"requests": 1842,
"prompt_tokens": 2190411,
"completion_tokens": 271624,
"ttft_ms_p50": 214,
"node_hours": 2.0
}Mortar runs
- Builds, releases and operates the control-plane and data-plane software
- Upgrades, observability and incident response under the access you grant
- Tenancy, routing, caching and scaling policy
You control
- The cloud account, regions, GPU quotas and billing
- IAM roles, KMS policies, networking and allowed endpoints
- Your data, model assets and application ingress
Use an OpenAI-compatible API for LLM inference
Point the OpenAI SDK you already use at a Mortar endpoint. No new client, no rewrite, the same code in Managed Cloud and BYOC.
from openai import OpenAI
import os
client = OpenAI(
base_url="https://<your-mortar-endpoint>/v1",
api_key=os.environ["MORTAR_API_KEY"],
)
reply = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "Hello, Mortar!"}],
)
print(reply.choices[0].message.content)import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://<your-mortar-endpoint>/v1",
apiKey: process.env.MORTAR_API_KEY,
});
const reply = await client.chat.completions.create({
model: "Qwen/Qwen2.5-7B-Instruct",
messages: [{ role: "user", content: "Hello, Mortar!" }],
});
console.log(reply.choices[0].message.content);curl https://<your-mortar-endpoint>/v1/chat/completions \
-H "Authorization: ******" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [{"role": "user", "content": "Hello, Mortar!"}]
}'Try Mortar's OpenAI-compatible LLM playground
Streaming over an OpenAI-compatible endpoint, with the serving telemetry Mortar records for every request alongside it.
Prompts run once you sign in, and the one you write comes with you. Sign in