How to size dedicated LLM inference for steady production traffic

Reserved inference is only cheaper than per-token pricing when you keep it busy. That makes sizing the whole game: too small and you are back to rate limits, too large and you are paying for idle GPUs. This post is the planning exercise, from the metrics you need to the comparison at the end.

It assumes you have already decided that dedicated LLM inference is worth evaluating. If you are still deciding, the framework post covers the signals.

Start from tokens, not requests

Requests are the wrong unit. A request can be a 200-token chat turn or a 60,000-token agent step with a full repository in context, and the serving cost of those two differs by orders of magnitude. Capacity for inference is measured in tokens per second, and you need it in two separate numbers:

• Prompt (input) tokens per second. These are processed in the prefill phase, which is compute-bound and fast per token.
• Completion (output) tokens per second. These are generated one at a time in the decode phase, which is memory-bandwidth-bound and slow per token.

A workload that is 95% prompt tokens, like document extraction with short structured outputs, stresses a GPU very differently from one that is 50% completion, like open-ended chat. Two workloads with identical total token counts can need very different reservations. Measure both.

If your current provider's usage dashboard only shows daily totals, add logging at the application layer. Every response from an OpenAI-compatible API carries a usage object with prompt and completion counts. Record it with a timestamp per request. A week of that data is enough to plan from.

Find the shape of the week

With per-request token counts in hand, aggregate into one-minute buckets and plot both series over at least one full week. You are looking for three numbers on each:

1. The floor. The level you exceed during almost all working hours. This is the load a reservation covers efficiently.
2. The typical peak. The daily high on a normal day, not the launch day or the incident.
3. The extreme peak. The worst minute in the week.

The floor is what you size a reservation to. The gap between the floor and the typical peak is what you either cover with headroom or hand off to a shared endpoint. The extreme peak is what you need a plan for, which is almost never "reserve enough to absorb it."

Weekly seasonality matters more than most teams expect. A coding agent product used by engineering teams will have a floor on weekdays and almost nothing on weekends. A consumer chat product might be the reverse. If the floor drops to near zero for two days out of seven, that is roughly 28% of the reservation idle, and it needs to be in the comparison.

Convert tokens per second into capacity

A reservation is described in terms of hardware: a number of GPUs of a given type, running a given model. Turning your tokens-per-second targets into a GPU count requires knowing what one unit of that hardware delivers for your model and your traffic mix. There is no universal answer, because throughput depends on:

• the model size and quantization
• the ratio of prompt to completion tokens
• the typical context length, since longer contexts reduce how many sequences fit in a batch
• the concurrency level, since batching improves throughput up to the point where it starts hurting latency

The honest way to get the number is to measure it, which is what LLM inference benchmarks exist for. When you read published results, check that the prompt length, output length, and concurrency match your workload before you plan against them. A throughput figure measured on short prompts will overestimate what you get with long-context agent traffic.

Once you have throughput per unit for something close to your mix, the arithmetic is simple: divide your target tokens per second by the per-unit throughput and round up.

Choose headroom deliberately

Sizing to exactly the floor leaves no room for the daily peaks, and it leaves no room for the throughput loss that comes with running a GPU at full saturation, where latency climbs sharply. Headroom is a product decision, and it has three inputs:

Latency targets. Throughput and latency trade off. A reservation running at 90% of its maximum throughput will have much worse tail latency than one at 60%. If your product is interactive, you want to operate well below the ceiling. If it is batch, you can run closer to it.

Peak-to-floor ratio. If your typical daily peak is 1.5x the floor, covering it with headroom is cheap. If it is 4x, covering it with reserved capacity means the reservation is mostly idle, and you are better off with a hybrid that spills the peak to a shared endpoint.

Growth. A reservation is a commitment over some term. Estimate where the floor will be at the end of that term, not the beginning, and decide whether to size for it now or plan to expand.

A common starting point for interactive workloads is a reservation that covers the typical peak at roughly 60 to 70% utilization, with anything above that spilling over. For batch workloads that can tolerate queueing, sizing closer to the floor and letting the queue absorb peaks is often the more economical choice.

Run the comparison

Now you have two costs to compare over the same period:

• Shared: total prompt tokens times the input price, plus total completion tokens times the output price, using the token counts you logged.
• Dedicated: the reservation cost for the capacity you sized, over the same period, regardless of tokens.

The LLM inference cost calculator does this arithmetic against Morph's pricing and lets you move the inputs around, which is useful for seeing how sensitive the answer is to utilization. Two adjustments make the comparison more honest:

First, add the shared cost of whatever traffic spills above the reservation in the hybrid case. Second, include the token cost of retries. On a shared endpoint at peak, some fraction of requests are retried after rate limiting, and every retry is billed. On reserved capacity with proper headroom, that fraction is close to zero.

The result is a utilization threshold: the fraction of the reservation you need to fill for it to cost less than the same tokens on the shared endpoint. If your measured floor puts you comfortably above that threshold, the reservation pays for itself. If it puts you below, either the workload is not steady enough yet or the reservation is oversized.

Things the spreadsheet does not capture

A few factors do not fit neatly into the cost comparison but belong in the decision:

• Filling spare capacity is free. Once a reservation exists, running an extra verification pass, a reviewer model, or a nightly re-index costs nothing at the margin. Teams often find uses for the headroom that improve the product.
• Predictable throughput has planning value. Knowing a batch job will finish in a fixed window, every night, is worth something to the people who depend on its output.
• Evaluation stability. Comparing prompts or model versions on a serving layer that is not shared with other tenants removes a source of noise.
• Operational load. You are taking on some monitoring and capacity management that the shared provider did for you. It is not much, but it is not zero.

A worked sequence

Putting it together, the planning loop looks like this:

1. Log prompt and completion token counts per request for one representative week.
2. Aggregate into one-minute buckets. Identify the floor, typical peak, and extreme peak for each series.
3. Get per-unit throughput for your model at a prompt length, output length, and concurrency that match your traffic, from benchmarks or your own load test.
4. Size a reservation to your chosen headroom policy.
5. Price it against your logged tokens at shared rates, including spillover and retries.
6. Re-run the exercise a month after switching. Real utilization is the number that matters, and it is the one you can only see once you are running.

For workloads driven by agents, step one often reveals a floor that is higher than anyone assumed. A fleet of coding agents working in VMs runs around the clock and every task turns into dozens of long-context calls. That is exactly the traffic profile that makes the arithmetic favor a reservation, and the coding agents post goes into why.