LLM inference for coding agents: why steady, long-context traffic favors dedicated capacity

A coding agent is not a chatbot with a code formatter. Its inference traffic looks different in almost every dimension that matters to a serving stack, and teams that plan capacity as if it were chat traffic tend to discover the difference in production. This post is about that traffic profile and what it implies for choosing between shared and dedicated LLM inference.

What a coding agent's traffic looks like

Take one task: "add rate limiting to the upload endpoint and write tests." A capable agent will read the relevant files, search for existing middleware, make an edit, run the tests, read the failure, edit again, run again, and eventually open a change. Each of those steps is at least one model call. Thirty calls for one task is ordinary; a hundred is not unusual for something involved.

Now look at the shape of each call:

• Long prompts. The context carries the system prompt, the task, the conversation so far, and whatever files and tool output the agent has pulled in. Tens of thousands of prompt tokens per call is typical, and the prompt grows as the task progresses.
• Short completions. Most steps produce a tool call or a small edit. A few hundred output tokens is common; a few thousand is the exception.
• Sequential dependence. Step N+1 cannot start until step N returns and its tool has run. Latency is additive across the whole task.
• Heavy prefix overlap. Consecutive calls share most of their prompt, since the conversation only grows at the end.
• Continuous load. Agents do not keep office hours. A fleet working through a queue of tasks in VMs produces traffic all day, with a floor that only drops when the queue empties.

Each of these has a consequence for serving.

Consequence 1: prefill dominates

With a prompt-to-completion ratio that can exceed 50 to 1, the serving cost of coding-agent traffic is overwhelmingly prefill. That has two effects.

On a shared endpoint, your requests are the expensive ones in the batch. Long prompts take more compute per request and reduce how many sequences fit alongside them, which is why agent traffic tends to see rate limits sooner than its request count would suggest.

On reserved capacity, prefill throughput becomes the number to size against, and it is worth checking specifically. LLM inference benchmarks that report throughput at long prompt lengths are far more relevant here than ones measured on short chat turns. When you run your own, use prompts from the top half of your length distribution, not the median.

Consequence 2: prefix caching is worth a lot

Because consecutive calls share a long common prefix, a serving stack that caches the key-value state of that prefix can skip most of the prefill work on every call after the first. The savings are largest exactly where coding agents spend their tokens.

Prefix caching is a serving-side feature, and its effectiveness depends on the cache actually being warm when your next request arrives. On a shared pool, your prefixes compete with everyone else's for cache space, and a request routed to a different replica finds a cold cache. On dedicated capacity, the cache holds your prefixes and nothing else, and routing is under your control. This is one of the more concrete reasons agent workloads see a larger benefit from isolation than the raw throughput comparison suggests.

Whether a given endpoint supports prefix caching, and how it exposes it, is something to confirm rather than assume. Check the documentation and verify it with a benchmark that sends the same prefix repeatedly and measures TTFT on the second and later calls.

Consequence 3: tail latency compounds

If one call has a p99 TTFT of four seconds, a thirty-step task has a real chance of hitting that tail several times. The task's latency distribution is far wider than any single call's. Users experience this as agents that are usually fine and occasionally take three times as long for no visible reason.

The fix is not lower average latency. It is lower variance, and variance on a shared endpoint is mostly other tenants' traffic. Reserved capacity makes your latency distribution a function of your own load, which you can measure, predict, and hold below the level where the tail grows. The benchmarking post describes how to find that level with a concurrency sweep.

There is a secondary benefit for debugging. When a task is slow on isolated capacity, the cause is in your system: a prompt that grew too large, a tool that returned too much, a step that looped. On a shared endpoint, the first question is always whether it was you or the pool, and that question is expensive to answer.

Consequence 4: the floor is high

Teams frequently underestimate their agent traffic floor because they think in terms of active users rather than active tasks. Ten engineers using an agent does not mean ten concurrent conversations. It can mean fifty tasks in flight, each generating a call every few seconds, across a workday that spans time zones.

Plot your tokens per second over a week, as described in the capacity planning post, and the floor is often the surprise. A high floor is the condition that makes a reservation economical, because the cost is flat and the capacity is filled. Run the numbers through a LLM inference cost calculator with your measured prompt and completion counts, and pay attention to what the prompt-heavy ratio does to the comparison.

Consequence 5: throughput guarantees change what you build

Once you know the capacity is there, certain product decisions become cheap that were expensive on a metered endpoint:

• Running a second model as a reviewer on every proposed change.
• Letting the agent run the full test suite and read all of the output instead of a truncated slice.
• Spawning parallel attempts at a hard step and picking the best.
• Re-planning from scratch when the agent gets stuck instead of pushing through.

Each of those is more tokens. On a per-token bill, each is a line item to justify. On a reservation with headroom, they are free until the headroom is gone, and they tend to make the agent better.

Practical setup

For a team running coding agents at steady volume, a reasonable architecture is:

1. Reserved capacity sized to the weekday floor with enough headroom to hold p99 TTFT at your target through the typical daily peak.
2. A shared endpoint behind the same OpenAI-compatible client as the spillover path, so that traffic above the reservation degrades to metered rather than failing. Because the API shape is the same, routing is a base-URL decision, not a code path.
3. Prefix-aware request routing if your reservation has more than one replica, so consecutive calls from one task hit the same warm cache.
4. Per-task latency budgets enforced by the orchestrator, so a task that hits the tail repeatedly is surfaced rather than silently slow.
5. Token accounting per task, not just per request, so you can see which agent behaviors drive load.

Where the agents run matters too

Inference is half of the loop. The other half is the environment where the agent's tool calls execute: reading files, running tests, installing dependencies, starting servers. That environment has its own latency and isolation requirements, and it is why we build Freestyle VMs as full Linux machines that start fast and can be forked, paused, and resumed. A fast model behind a slow sandbox is still a slow agent. When both halves are predictable, the task-level latency becomes something you can actually put a number on.

For the other side of the traffic spectrum, high-volume chat, extraction, and scheduled workflows, the migration post covers what changes.