Notes on the machinery

1 · The thing that was annoying

The application I was working on before this one had model identifiers in forty-odd places. Not in a config file — in the call sites, next to the prompt, because that is where they end up when each one is added by whoever needed it that week. Changing which model served a feature meant a code change, a review and a deploy. Nobody could say what any single feature cost, because the only cost signal was one invoice a month with one number on it. And when quality dropped — a model revision, a prompt that stopped fitting — the first report came from a support ticket.

Three separate problems, and they have one shape: the decision about which model serves a request is made at authoring time, in code, by somebody who cannot see what it costs or whether it worked. So the design here moves that decision to request time, to one place, and makes it report on itself. A capability such as text.summarize is a row in a table rather than a constant in a file — publish one from the console and it serves calls immediately, with no deploy. What the caller sends is the outcome it wants. What serves it is ours to decide, and ours to be wrong about in public.

2 · Why the router reads the request

The obvious design is a route table: this endpoint uses the cheap model, that one uses the expensive one. It is simpler, it is inspectable, and it needs no classification pass. I rejected it, and the reason is narrower than “semantics are better”.

A route table binds cost to the endpoint. But an endpoint is not a workload. One capability emits several kinds of work on the way to one answer: a research call rewrites the query, plans retrieval, compresses evidence and then synthesises — four steps that want four different models. Bound to an endpoint, all four get whatever the most demanding of them needed, and the cheap steps are subsidising the expensive one at every call. Splitting the endpoint fixes it only until the next workload arrives, at which point you are maintaining a table whose rows are guesses about traffic you have not seen. The table is never wrong loudly. It is wrong quietly and forever.

So the routing unit here is the task class — what kind of work this call is — and it is deliberately a different axis from the capability. Billing keys on the capability, because that is what was bought. Statistics key on the task class, because that is what varies. Each request is classified by domain, task type and complexity, and complexity picks one of five effort tiers, cheapest first.

That has costs, and they are worth naming. Every request pays for a classification pass. The edges between tiers are a threshold someone has to tune, and a threshold moves money: two prompts either side of an edge get different models and different bills for a reason no user could predict. And a classifier is confidently wrong in the way that costs you — not a dramatic misroute, but a steady under-read of requests phrased in vocabulary it has not seen. Those are the trade, and the trace on the front page shows the tier each request was read as so that a bad read is visible rather than inferred.

3 · What the spend check guarantees

The cost of a call is estimated before it is made, not reconciled after. The estimate counts the prompt, the capability’s system prompt — which is billed as input on every call and would otherwise be discovered at settlement — and a share of the answer the chosen tier is likely to produce. That estimate is converted to user tokens and checked against the tenant’s balance above the provider call. An unfunded request is refused before anything is spent; the classification pass is the only work it costs.

The deduction itself happens after the provider returns, against real observed usage rather than the estimate. So a tenant is charged for what the answer actually cost, and the estimate only ever decides whether the call may start.

It is a check, not a contract. Concurrency is bounded per tenant, and the check runs on every call, but I would not describe it as a hard cap in a contract until the hold is reserved and released atomically. That work is in section 7.

4 · What the answer check does and does not catch

Every answer is checked against the original request before it is returned. The standard objection is immediate and correct: a quality layer that is a second model call is a second cost and a second latency hit on every request, and it can eat the entire saving the router just produced.

So the check is not a model call. It runs in the gateway, deterministically, and costs no provider spend. It catches the failures that have a shape:

• Refusals. A model that declined is scored as a refusal rather than passed through as prose.
• Truncation. An answer cut off at its length limit is flagged.
• Reductive tasks that reduced nothing. A summary that is barely shorter than its source has not summarised it, however well written it is.
• Goal coverage. Each goal recorded at the start is checked against the answer, and a goal the answer never touches is named in the response.

And here is what it does not catch, which matters more. A fluent, correctly shaped, confidently wrong answer passes. Every one of those checks is structural. None of them is a claim about whether the answer is true, and the page does not represent it as one.

The design decision that earns its keep is not the checks — it is where they read from. What the request was asking for is written into a journal at classification time, before any answer exists. The check claims it back by the request’s ID and never receives it from the code that produced the answer. If the answering path could hand it the goals, it would be checking the pipeline against itself, and it would agree with itself every time. When that handoff is lost, the answer warns rather than fails: the tenant has already been charged, and a platform fault is not a bad answer.

5 · Slow is harder than down

A provider returning an error is easy. The interesting failure is a provider that answers in forty seconds instead of four, because every layer above it is still waiting and nothing anywhere has registered an error.

Streamed and unary calls get different patience, for one reason: a streamed turn signals that it is alive with its first token, so a long one can be given room, while a unary call’s only signal is completion, so a long one is indistinguishable from a hang.

Failures are contained by circuit breakers, scoped so that a vendor having a bad afternoon does not trip capabilities served by somebody else. They are the standard three states with one addition I would argue for: an operator override is a fourth state rather than a boolean flag beside the other three. As a flag, an automatic transition silently undoes an operator’s decision the next time a probe succeeds, and the audited override is then neither audited nor an override. A model an operator switches off leaves the eligible set on the next call, and routing picks the cheapest of what remains.

6 · What I got wrong

The router prices every candidate before choosing between them. The first version priced each one at its declared output ceiling — the largest response that model could possibly return. It is the conservative choice, it never under-quotes, and it was wrong in a way that took a while to see.

The obvious consequence: on an ordinary prompt returning a few hundred tokens, pricing at the ceiling overstates the spend several-fold, which refuses calls the tenant could comfortably afford.

The one I did not see coming: it was choosing the models. Selection is cheapest-first, and the estimate was dominated by a field that describes what a model is permitted to return, not what this request will make it return. So the router systematically preferred whichever model declared the smallest output ceiling, which is a property of the vendor’s configuration and has nothing to do with the request being served. Every call succeeded, every estimate was conservative, and the selection was being driven by a number nobody had thought of as a routing input. The estimate now scales with what the classifier already knows about the request.

The generalisable mistake is not “I was too conservative”. It is that I put a field into a ranking function without asking what it was a measurement of. A conservative estimate is a defensible position. A ranking driven by a field that does not describe the thing being ranked is a bug that produces plausible output forever.

7 · What is not built

• The spend hold is not yet atomic. Section 3. The balance check runs on every call; reserving and releasing the estimate as one operation is the next piece of work.
• The utility work is deterministic code. Classification, condensation and the answer check run in the gateway rather than as hosted model calls.
• No mid-request downgrade. A provider that goes slow is waited on; the breaker protects the next request, not this one.
• The answer check is structural, not semantic. Section 4. It does not detect a confident falsehood and is not represented as doing so.

If you find something on this page that is wrong, or a gap that should have been on this list, I would like to know: [email protected].