How to benchmark LLM inference before you commit to capacity

Inference benchmarks are easy to run and easy to get wrong. A single number like "tokens per second" hides the choices that produced it, and the choices are usually what decide whether the endpoint works for your product. This post is about running a benchmark whose result you can actually plan against, whether you are comparing shared endpoints or evaluating dedicated LLM inference before reserving it.

The four metrics that matter

Every serving stack is a two-phase process. Prefill reads the prompt and produces the first token. Decode generates the rest, one token at a time. Good benchmarks measure both phases separately, because they are bounded by different resources and fail in different ways.

Time to first token (TTFT). How long from sending the request until the first token arrives. This is the prefill phase plus queueing. It grows with prompt length and with load. For interactive products it is what users perceive as "did anything happen."

Time per output token (TPOT). The interval between successive tokens during decode. It is what users perceive as reading speed, and what determines how long a 500-token response takes after it starts. It degrades as concurrency rises, because more sequences share the same memory bandwidth.

Throughput. Total tokens per second the endpoint sustains across all concurrent requests. This is the capacity number you size reservations against. It rises with concurrency until the batch is full, then flattens, while TPOT keeps getting worse.

Tail latency. The p95 and p99 of both TTFT and end-to-end time. Averages hide the requests that time out, and in multi-step systems the tail is what compounds. Report percentiles, not means.

If a benchmark reports only one of these, it is not enough to plan from.

Make the test look like your traffic

The most common benchmarking mistake is testing with the wrong shape of request. Throughput measured on 100-token prompts and 100-token completions says almost nothing about a workload of 30,000-token prompts and 300-token completions. Before running anything, pull the following from your production logs:

• the distribution of prompt lengths, not just the mean
• the distribution of completion lengths
• the ratio of prompt to completion tokens overall
• the typical and peak concurrency
• whether requests stream, and whether they use tool calls or structured output

Then build a request set that matches. Use real prompts if you can, or synthetic ones with the same length distribution. A benchmark on representative traffic will produce a lower, more honest throughput figure than a benchmark on short prompts, and it is the honest figure you need.

When reading published LLM inference benchmarks, apply the same lens: look for the prompt length, output length, and concurrency the numbers were measured at, and weigh them accordingly. Numbers measured near your traffic profile are useful. Numbers measured far from it are a starting point at best.

Sweep concurrency, do not pick a point

Run the same request set at increasing concurrency levels: 1, 4, 16, 64, and upward until the endpoint saturates or starts returning errors. At each level, record all four metrics. You are looking for two things:

The knee in the throughput curve, where adding concurrency stops adding throughput. That is the endpoint's effective capacity for your traffic shape.

The latency budget crossing, where TTFT or TPOT exceeds what your product can tolerate. On most stacks this happens before the throughput knee. The concurrency level just below the crossing is the load you can actually run, and it is often well below the theoretical maximum.

The gap between those two points is your headroom. A stack with a wide gap degrades gracefully. A stack with a narrow gap goes from fine to unusable quickly, which matters a lot when you are deciding how close to the ceiling to operate.

Test the features you depend on

Compatibility is a claim about request and response formats. Behavior under those formats is something you verify. Before routing production traffic, exercise:

• Streaming. Confirm token deltas arrive incrementally and that the connection handles long generations without dropping.
• Tool calls. If your agents use function calling, check that the endpoint produces the same structure your parser expects, including parallel calls and edge cases like empty argument objects.
• Structured output. If you rely on JSON mode or schema-constrained generation, test with your real schemas, especially the large ones.
• Long contexts. Send prompts at the top of your length distribution. Some stacks handle them correctly but with much worse TTFT than the average suggests.
• Stop sequences, temperature, and sampling parameters. Confirm the ones you set are honored rather than silently ignored.
• Error semantics. Push past capacity deliberately and see what comes back: a clean 429 with a retry hint, a timeout, or a partial response. Your retry logic depends on this.

None of these show up in a throughput number, and any one of them can block a migration.

Measure stability over time, not just once

A ten-minute benchmark tells you about that ten minutes. Shared endpoints in particular vary with time of day, because their load is other tenants' load. Run the same test at several points across a day and a week, and plot the variance. Consistency is a feature, and it is one of the main things a dedicated reservation is buying you: your latency distribution becomes a function of your own traffic, so a test today predicts behavior tomorrow.

If you are benchmarking dedicated capacity, run the stability test at your planned utilization, not at idle. An empty reservation is fast. The question is how it behaves at 60 or 70% load for an hour.

A minimal harness

You do not need a sophisticated tool. A script that does the following is enough for a first pass:

1. Loads a request set sampled from production logs.
2. Fires requests at a fixed concurrency using an OpenAI-compatible client with streaming enabled.
3. Records, per request: send time, first-token time, last-token time, prompt tokens, completion tokens, and status.
4. Computes TTFT, TPOT, end-to-end time, and throughput, with p50, p95, and p99 for each.
5. Repeats across the concurrency sweep and writes one row per level.

Run it from the same region your production traffic originates in. Network latency to the endpoint is part of TTFT as your users experience it, and a benchmark run from a different continent will mislead you in both directions.

Open-source load generators exist for this, and several serving frameworks ship one. Whatever you use, the requirements are the same: representative requests, a concurrency sweep, percentiles, and enough runs to see variance.

Reading the results

A good result for your workload is not the highest throughput. It is the combination of:

• throughput at your latency budget that comfortably covers your typical peak
• a p99 TTFT that your product can hide behind a loading state
• a TPOT that reads as fluent if you stream to users
• low variance across the day
• correct behavior on every feature you tested

If a candidate meets those, the throughput figure at your latency budget is the number to take into capacity planning. It converts directly into how much reserved capacity you need, and from there into a cost you can compare against per-token pricing.

Why this matters more for agents

For a chat product, a slow p99 is one slow reply. For an agent, it is one slow step in a chain of many. A coding agent working through a task in a VM might make thirty model calls before it is done, and the task's latency is the sum of theirs. Tail latency that is tolerable per call becomes intolerable per task. That is why the concurrency sweep and the stability test are not optional for agent workloads, and why the isolation that reserved capacity provides tends to show up first in those products. The coding agents post covers that traffic profile in detail.