your inference engine evicts the KV cache the moment the agent calls a tool.

Then the tool returns. Then you recompute everything from scratch.

This is happening in every production agent deployment running on vLLM right now. Not occasionally. Every time an agent makes a tool call. The framework sees an idle GPU slot, evicts the cache to free memory for other requests, and when the agent resumes it pays full prefill cost again on a context it already processed.

The fix is embarrassingly obvious in retrospect. Nobody shipped it until a few months ago.

Let me make the problem concrete because it is easy to miss in profiling data.

A code agent receives a task. It calls the LLM: "analyze this repository, identify the bug, write a fix." The LLM processes 8,000 tokens of context -- the task, the file contents, the conversation history -- and produces a tool call: run_tests(patch_v1.py). That tool call kicks off a CI run. The CI run takes 45 seconds.

During those 45 seconds, the inference framework sees a request that hasn't produced a new token in 45 seconds. vLLM's scheduler sees an occupied KV cache slot that isn't being used. Another request is waiting. The scheduler evicts the cache.

The CI finishes. The test output comes back. The agent needs to continue. The LLM needs to see: the original 8,000 tokens of context, plus the tool result. That full 8,000+ tokens goes through prefill again. From nothing. Because the cache was evicted 40 seconds ago.

You paid twice for the same prefill. And the second payment happened at peak load, when another request was already waiting.

The per-request cost for multi-step agentic workflows isn't what your throughput benchmarks show. It's higher -- sometimes significantly higher -- because every tool call that takes longer than the scheduler's eviction threshold is a full prefill redo. At 8,000 tokens per context and 45 seconds per CI run, you are burning significant compute on work you already did.

Continuum (November 2025, updated January 2026, now in the vLLM preview branch) proposes a specific fix: give the KV cache a time-to-live based on predicted tool call duration.

The key insight is that tool calls are not uniformly slow. web_search averages 3 seconds. run_tests averages 45 seconds. read_file completes in milliseconds. The inference engine doesn't need to guess -- it can observe tool call durations in production and build a prediction model per tool type.

Continuum instruments the agent framework to log tool call start and completion times, builds a lightweight per-tool duration distribution (the paper uses a simple mean estimate that stabilizes quickly), and uses that prediction to set a TTL on the KV cache for each in-flight agent step. If the predicted tool call duration is under the TTL threshold, the cache stays alive. If the tool call is expected to take longer than it's worth keeping the cache warm, it's a candidate for eviction -- but with enough advance notice to make that decision deliberately rather than reactively.

The second piece: program-level scheduling. Continuum tracks agent workflow structure -- which steps are sequential, which are parallel, which tools can run concurrently -- and uses that to pipeline KV cache management with tool execution. While the slow tool is running, Continuum prefetches context for the next expected agent step into GPU memory. The tool finishes. The context is already there.

The result on SWE-Bench and BFCL with Llama-3.1 8B and 70B: measurable improvement in average job completion time compared to state-of-the-art baselines including InferCept and Autellix. More importantly, the improvement increases with the number of turns -- the more steps in an agent workflow, the more times the naive eviction policy fires, and the more Continuum's TTL-based approach saves.

The reason I want to write about this specifically is the framing error it reveals.

We built inference serving infrastructure for the request-response pattern. One request in, one response out, KV cache lives as long as the request is active, gets evicted when the response is complete. That pattern is correct for a chatbot. It is wrong for an agent.

Agents have a fundamentally different request lifecycle. An agent step is not a request that completes when the LLM produces a response. An agent step is a request that completes when the entire workflow episode finishes -- which includes tool calls, sub-agent invocations, external state updates, and potentially multiple LLM calls. The KV cache for an active agent episode is not the KV cache for a completed request. It is shared state for an ongoing process.

The serving frameworks were not designed for this. They were designed before agents were the dominant workload. The eviction policy that's optimal for isolated requests -- free the memory as soon as the token stream ends -- is actively harmful for agent workloads, because the token stream ending is not the end of the episode.

Continuum fixes this with a surgical change: TTL on cache retention, calibrated per tool type, predictively managed. It doesn't require a new serving architecture. It doesn't require changing the model. It requires instrumenting tool call durations and adding a TTL parameter to the eviction policy.

Code is in the vLLM preview branch right now.

There is a second problem this surfaces that Continuum doesn't fully solve: the KV cache is per-GPU-instance.

In a multi-node serving cluster, an agent's workflow might span multiple LLM calls, and those calls might land on different GPU instances depending on the load balancer. Each time the call lands on a different GPU, the cache miss is guaranteed regardless of TTL -- the cache from the previous step is on a different machine.

This is the routing problem for agents. It's distinct from the routing problem for single-request sessions. For sessions, you can use prefix-caching-aware routing to preferentially direct requests to the GPU that has the relevant prefix cached. For multi-step agent workflows, you need to ensure that every step in an episode lands on the same GPU instance, or you need a distributed KV cache that can transfer state between GPUs fast enough that the miss is cheaper than recompute.

llm-d (IBM/Google/Red Hat) is building cluster-level KV cache tracking to enable this -- a global index of which GPU instance holds which KV cache blocks, updated in real time via KVEvents, used to route agent steps to the instance that already holds the relevant context. The data-to-metadata ratio is 1,000,000:1 -- the index overhead is negligible even at large cluster scale.

The combination of Continuum's TTL-based retention and llm-d's cluster-level routing is the complete answer to the agent KV cache problem. Neither alone is sufficient.

the eviction policy was designed for chatbots.

you are running agents.

the agent calls a tool. the framework evicts the cache. the tool returns. you pay full prefill cost again.

every time. on every tool call longer than your eviction threshold. at production load.

instrument your agent framework. measure the gap between first-prefill cost and re-prefill cost across tool calls. the number you find is the compute you are burning on work you already did.

P.S. The per-tool TTL calibration gets more accurate over time as you collect real duration data from your own tool implementations. The paper shows the mean estimate stabilizes within a small number of observations per tool type. This means the system improves automatically as agents run in production -- the inference overhead for frequently-called tools goes down as the model's duration estimates tighten. You get better performance without changing anything. That is an underrated property of the design.