> ## Documentation Index > Fetch the complete documentation index at: https://vetta.sh/docs/llms.txt > Use this file to discover all available pages before exploring further. # Model router & inference > How Vetta routes each model call to inference — an agent's turn or one you make yourself — what the completion window changes about it, and how model tokens are metered. Every model call goes through Vetta's **model router** — the ones an agent makes inside a session, and the ones you make yourself through the [model proxy](#calling-the-router-directly). The router picks the inference backend for each call based on two inputs — the **model** and the [**completion window**](/docs/concepts/completion-window) — meters the tokens against the caller's balance before the call runs, and returns the result. Inside a session the [harness](/docs/how-vetta-is-built#the-three-layers) decides *when* to call the model; the router — part of the [runtime](/docs/concepts/runtime) — decides *where the call goes* and *what it costs*. You never address a backend directly. You name a model and a window; the router does the rest. There are two ways to name them: * **On an agent** — `model` and `window` are fields of the [agent](/docs/concepts/agents), overridable per session. Every turn the harness runs goes through the router with them. * **Directly** — one call, no agent and no session, through the [proxy](#calling-the-router-directly) in the Messages or Chat Completions format. Both use the same catalogue, the same lanes and the same meter; only who pays differs. Images and video are **not** model calls: they have their own catalogue and their own pages — [Images](/docs/api/images) and [Video](/docs/api/video). ## Two inference lanes Vetta federates two inference backends behind one interface: * **Aggregated inference network** — a broad catalogue of models available at interactive latency. This is the default lane and serves every `immediate` request. On this lane the router asks the network for the **fastest** provider that hosts the model, not the cheapest — `immediate` is the pay-for-speed window, and the default price-ordered routing was measured putting a cold first token tens of seconds out on a slow node. One consequence is worth stating plainly: because a model is hosted by several providers at different prices, **two identical `immediate` calls to the same model id can cost different amounts** depending on who served them. You are billed what the call actually cost, so the difference is real rather than an estimate — see [What a model call costs](#what-a-model-call-costs). The `priority` and `loose` windows are unaffected. * **Completion-window pool** — specialized open-weights hosting that offers genuine reduced tariffs for the `priority` and `loose` windows at reduced latency cost. This lane serves `priority` and `loose` requests, and it only hosts a **specific set of window-supported models**. ``` model call (model + window) │ ├─ window = immediate ─────────────▶ Aggregated inference network │ any model in the catalogue · immediate tariff │ └─ window = priority | loose ──────▶ Completion-window pool │ priority / loose tariff │ └─ model NOT window-supported ─▶ 400, refused before any spend ``` ## The routing rule The window determines the lane, and the lane constrains the model: | Window | Lane | Model requirement | | ----------- | ---------------------- | --------------------------------------------------- | | `immediate` | Aggregated network | Any model in the catalogue, including `vetta/auto`. | | `priority` | Completion-window pool | Must be a **window-supported** model. | | `loose` | Completion-window pool | Must be a **window-supported** model. | A **non-default window** (`priority` or `loose`) with a model that the completion-window pool does not host is refused with a typed error — `window_unavailable`, **HTTP 400** — before any inference runs and before any spend. `immediate` always works, on any catalogued model. See [Errors](/docs/api/errors). This is a deliberate fail-closed: rather than silently downgrading a `loose` request to the interactive tariff (and quietly overcharging you), the router rejects the combination so you fix it explicitly — either pick a window-supported model, or drop to `immediate`. ## The catalogue is live There is **no curated list**. The catalogue is read from the inference network at request time and cached briefly, then filtered down to the models Vetta can actually drive an agent with: * text out — a model that returns images or audio is not an agent's model; * tool calling — the harness cannot run a loop with a model that cannot call a tool; * a published per-token price for both prompt and completion, because a model that cannot be quoted cannot be metered, and metering before the call is what makes spend fail closed. That is **hundreds of models**, and it moves on its own: a model the network publishes today is runnable today, with no release of ours. Two consequences follow for anything that reads it — the listing is **paged**, and it is **searchable**. Nothing should assume one call returns the whole catalogue, and nothing should hard-code a model id it has not checked. ## Discovering models `GET /v1/models` is the catalogue; `GET /v1/models/{id}` is one entry, for the id you already hold. ```bash CLI theme={"system"} vetta models list --search glm --limit 5 vetta models list --window priority # only what that window can serve vetta models get zai-org/GLM-5.2-FP8 ``` ```bash API theme={"system"} curl -fsSL "https://api.vetta.sh/v1/models?search=glm&limit=5" \ -H "authorization: Bearer sk_live_..." # Model ids contain a slash — percent-encode the path segment. curl -fsSL "https://api.vetta.sh/v1/models/zai-org%2FGLM-5.2-FP8" \ -H "authorization: Bearer sk_live_..." ``` ```typescript TypeScript theme={"system"} const page = await vetta.models.list({ search: "glm", limit: 5 }); const more = await vetta.models.list({ after: page.next_cursor }); const m = await vetta.models.retrieve("zai-org/GLM-5.2-FP8"); console.log(m.supported_windows); // ["immediate","priority","loose"] ``` | Query | What it does | | -------- | ------------------------------------------------------------------------------------------------ | | `window` | Narrow to the models that window can actually be served in — `immediate`, `priority` or `loose`. | | `search` | Free text, matched against the model's id and name. | | `limit` | Page size, 1–100. Defaults to 20. | | `after` | The previous page's `next_cursor`. | The listing is [cursor-paged](/docs/api/pagination) like every other list on the API: `has_more` and `next_cursor` are real, not constants. An id the deploy does not serve is a `not_found` (404) from the by-id route — which makes it the cheapest way to validate a model id before you run on it. ```json theme={"system"} { "object": "model", "id": "zai-org/GLM-5.2-FP8", "context_window": 1048576, "max_output_tokens": 32768, "supported_windows": ["immediate", "priority", "loose"], "efforts": ["low", "medium", "high"] } ``` The model object publishes **no prices** — the same line [`GET /v1/media/models`](/docs/api/media) draws. What you spend is bounded by the agent's [budget](/docs/concepts/budgets) before the call, and read back as actuals from [`vetta agent spend`](/docs/cli/agents#spend). A published rate card would be a number to reconcile against; the ledger is the number that is true. `?window=` narrows to what a window can serve — the same derivation the router refuses on, so a model listed for a window is never rejected for it. ## `max_output_tokens` Every entry publishes `max_output_tokens`: the longest reply that model may produce, taken from what the model itself advertises and clamped to a platform ceiling. It is **per model**, not one number for the fleet — a model that can write 131 072 tokens and one that can write 8 192 are not bounded the same way. It is also what the **pre-flight quote is bounded by**. Before a call runs, the router reserves the worst case: every input token at the input rate, plus `max_output_tokens` at the output rate. The call then settles at what it actually used and the remainder is released. So `max_output_tokens` sets how much of a [budget](/docs/concepts/budgets) one in-flight call reserves, not what it costs — a long-output model holds more credit while it is running, and returns the difference when it is done. ## Vetta Auto `vetta/auto` is a model id like any other, and it picks the model per request: you name the task, the router picks the model that fits it, call by call. It is useful when a workload is uneven — a mix of trivial and hard turns — and you would rather not pin one model expensive enough for the worst of them. ```typescript TypeScript theme={"system"} const agent = await vetta.agents.create({ name: "Triage", model: "vetta/auto", budget: { capMicroUsd: "25000000", maxTaskMicroUsd: "3000000", period: "month" }, }); ``` Two things are specific to it: * **`immediate` only.** It routes across the aggregated network, so it never runs in the completion-window pool. `priority` or `loose` with `vetta/auto` is refused with `window_unavailable`, exactly like any other unsupported pair. * **It is priced as a ceiling, then settled at the model that answered.** Because the model is not known until the call is routed, there is no rate card to quote from. So the request carries a **hard price cap** — \$5 per million input tokens and \$25 per million output tokens — which the network enforces: a call it cannot serve inside the cap is refused rather than routed to something dearer. The pre-flight hold is taken at exactly those cap rates, so it is a genuine upper bound. The debit is then settled at the **rate of the model that actually answered**, which is normally well below the cap. You are never billed above the ceiling you were quoted. `vetta/auto` selects a different model for different requests by design. Pin a specific id instead when a run has to be reproducible, or when a prompt is tuned to one model's behaviour. ## Calling the router directly You do not need an agent to use a model. The **model proxy** is the router with nothing in front of it: one request in, one reply out, billed to the organization. It exists for the call a session is the wrong shape for — a classifier in your own pipeline, a one-off summary, an existing client you would rather repoint than rewrite. Two doors, one router: | Door | Route | Format | | ---------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Messages | [`POST /v1/proxy/anthropic/v1/messages`](/docs/api/proxy#post-v1proxyanthropicv1messages) | The Messages request and response, `system` and `tools` included, streamed or not. | | Chat Completions | [`POST /v1/proxy/openai/v1/chat/completions`](/docs/api/proxy#post-v1proxyopenaiv1chatcompletions) | The Chat Completions envelope — `choices`, `finish_reason`, `usage`, `data:` chunks when streamed. | Any id the catalogue publishes goes in `model`, `vetta/auto` included; the window travels as the `Vetta-Window` header because neither format has a field for it, and unset means `immediate`. ```typescript TypeScript theme={"system"} const reply = await vetta.proxy.messages( { model: "vetta/auto", max_tokens: 512, messages: [{ role: "user", content: "Is this ticket urgent? …" }] }, ); const chat = await vetta.proxy.completions( { model: "zai-org/GLM-5.2-FP8", messages: [{ role: "user", content: "Summarise this invoice." }] }, "priority", ); ``` ```bash CLI theme={"system"} vetta proxy message --model vetta/auto --text "Is this ticket urgent? …" vetta proxy url # the base URL to point an existing Messages-format client at ``` ```bash API theme={"system"} curl -fsSL https://api.vetta.sh/v1/proxy/openai/v1/chat/completions \ -H "authorization: Bearer sk_live_..." \ -H "vetta-window: priority" \ -H "content-type: application/json" \ -d '{ "model": "zai-org/GLM-5.2-FP8", "messages": [{ "role": "user", "content": "Summarise this invoice." }] }' ``` What a direct call shares with an agent's, and what it does not: | | Agent turn | Direct call | | -------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------- | | Catalogue, lanes, `window_unavailable` | same | same | | Pre-flight quote → stream → settle at actual usage | same | same | | Who pays | the agent's [budget](/docs/concepts/budgets), then the org | the org balance directly — `insufficient_credits` when the quote does not fit | | Ledger entry | carries `session_id` and `agent_id` | carries the request id (`x-request-id`) and neither | | Tools | run by the harness inside the sandbox | returned to you as `tool_use` / `tool_calls`; you run them and send the result back | | State between calls | the session's durable log | none — send the whole conversation every time | | Scope | `sessions:write` | `proxy:write` (or `sessions:write`), see [authentication](/docs/api/authentication#scopes) | A direct call carries **text, tools and streaming** — and refuses, as `validation_failed`, what it cannot carry: sampling and decoding controls (`top_p`, `top_k`, `stop`, `seed`, `n`, `logprobs`, penalties), structured output (`response_format`), `tool_choice`, extended-thinking blocks, and image or document content parts. Nothing is dropped on the floor; a setting the router will not honour is a call you should not pay for. The per-field detail is on the [proxy reference](/docs/api/proxy). ## Choosing which model answers `vetta/auto` chooses inside the call, from the whole network. There is also an **optional add-on** that chooses from **your** shortlist, with **your** descriptions: name two to sixteen catalogue ids, say in a line what each is for, and an evaluation model reads the conversation and says which one should answer it. It comes in two shapes. The first hands you the id, so you can log it or branch on it: ```ts theme={"system"} const pick = await vetta.models.autoSelect({ options: { "anthropic/claude-sonnet-4.5": "Routine coding, edits and tool use", "openai/gpt-5.6-sol": "Hard multi-step reasoning or architecture", "google/gemini-3-flash": "Short factual answers where speed matters", }, messages, default: "anthropic/claude-sonnet-4.5", min_confidence: 0.5, }); // pick.model is a concrete catalogue id — pass it as `model` on the next call. ``` The second is the same policy riding on the call itself, so the router makes the pick and the model call in one step — per call on either proxy door, or per turn on an agent: ```ts theme={"system"} const reply = await vetta.proxy.messages({ model: "anthropic/claude-sonnet-4.5", // the default; must be one of the options model_selection: { options: { /* as above */ }, min_confidence: 0.5 }, max_tokens: 1024, messages, }); reply.model; // the id that answered ``` Everything after the pick is the ordinary path above: the chosen id is resolved, quoted from **its** rate card for the request's window, admitted, routed and settled. It is an add-on, not a change to routing — `vetta/auto`, pinned ids, windows and the catalogue behave exactly as described on this page whether or not you use it. Full detail — the option rules, what `default` and `min_confidence` actually do, what a selection costs and adds in latency, and how it behaves when the evaluator is unavailable — is on [Model auto-selection](/docs/concepts/model-selection). ## What a model call costs Model spend — an agent's or a direct call's — is metered per token across a **five-tier ledger**: each class of token is priced separately because the backends bill them separately. The request's [completion window](/docs/concepts/completion-window) tariff sets the rate the call is quoted and reserved at; where the backend reports what the completion actually cost, that figure is what you are billed from, so the amount tracks the provider that served the call rather than a list price for the model id. | Token tier | Field | What it is | | --------------- | ------------- | ------------------------------------------------------------------------- | | **Input** | `input` | Fresh prompt tokens sent to the model. | | **Cache write** | `cache_write` | Prompt tokens written into the provider's prompt cache. | | **Cache read** | `cache_read` | Prompt tokens served from cache — billed at a fraction of the input rate. | | **Output** | `output` | Tokens the model generates. | | **Reasoning** | `reasoning` | Internal thinking tokens, when the model and `effort` produce them. | These five fields are the canonical token ledger: every inference debit records all five, and they are the components a [session's](/docs/concepts/sessions#cost-and-usage) `token_usage` and the billing line items are built from. Cache-read tokens are the reason list-price rate cards mislead. A card that bills every input token at the full input rate over-states real cost by **1.017×–4.176×** depending on harness and window (see [Benchmarks](https://usenaive.ai/benchmarks)). Vetta meters each tier at its real rate, so your bill tracks what the backend actually charged — not a rate-card fiction. Because every call is priced against these five tiers *before* it runs, a call that would breach your [budget](/docs/concepts/budgets) — or, for a direct call, your balance — is refused rather than discovered on an invoice. Model spend shows up as the `model` line item in the agent's spend breakdown: ```bash CLI theme={"system"} vetta agent spend Refunder --by component ``` ```json theme={"system"} { "by_component": { "model": 11902000, "computer": 481000, "search": 6474, "media": 390000 } } ``` Amounts are integer micro-USD (`11902000` is \$11.902). The `model` component is the sum of the five token tiers above; `computer`, `search` and `media` are the other components a debit can carry, and a component with no spend is absent rather than zero. ## Effort Some models accept an `effort` level that trades latency and reasoning-token spend for quality. Effort is independent of the completion window: the window sets the *tariff and latency lane*, effort sets *how hard the model thinks within it*. Which levels a model accepts is published per model in `efforts`, read from what the model itself advertises and narrowed to the three wire values. An empty `efforts` means the model takes no effort setting at all — most do not — so check the entry before you pin one. ```typescript TypeScript theme={"system"} const agent = await vetta.agents.create({ name: "Deep", model: { id: "zai-org/GLM-5.2-FP8", effort: "high" }, window: "priority", budget: { capMicroUsd: "25000000", maxTaskMicroUsd: "3000000", period: "month" }, // $25 / $3 }); ``` ## Configuration reference A model ID (e.g. `zai-org/GLM-5.2-FP8`, or `vetta/auto` to pick per request) or an object `{ id, effort }`. Set on the agent and overridable per [session](/docs/concepts/sessions#override-agent-configuration-for-a-session). Any id [`GET /v1/models`](#discovering-models) publishes is legal; anything else is `validation_failed`. The model identifier. `low`, `medium`, or `high`, for models that support it. The [completion window](/docs/concepts/completion-window). `immediate` routes to the aggregated network; `priority` and `loose` route to the completion-window pool and require a window-supported model. ### Read-only model fields Maximum context length in tokens. The longest reply this model may produce, and what the [pre-flight quote](#max_output_tokens) is bounded by. Which windows the model can run in. `immediate` is always present; `priority`/`loose` appear only for pool-hosted models. Effort levels the model accepts — `low`, `medium`, `high`, or empty when it accepts none. The three-price model, and why you choose it per request.